checkers: add alua path checker
Subsystems (2)
cover checkers: add alua path checker
For ALUA-capable storage arrays, multipath-tools currently uses TUR (Test Unit Ready) as the default path checker while a separate prioritizer determines ALUA state. When sysfs provides ALUA state (the common case), no RTPG command is needed for priority. However, if sysfs is unavailable or detect_prio is off, the prioritizer issues RTPG, resulting in two SCSI commands per path check cycle: TUR + RTPG.
This patch introduces an 'alua' path checker that uses RTPG as the path check command, providing two benefits:
1. For configurations where RTPG is needed for priority, this eliminates duplicate I/O by combining path checking and ALUA state retrieval into a single RTPG command.
2. RTPG provides richer path state information than TUR:
* TRANSITIONING state detected and mapped to PATH_PENDING, preventing false all paths down during controller failover
* STANDBY state detected and mapped to PATH_GHOST
* UNAVAILABLE/OFFLINE states properly distinguished
The richer state information is valuable even when sysfs is available, as TUR cannot distinguish between these ALUA-specific conditions.
This design intentionally couples the checker and prioritizer: the alua_cached prioritizer consumes state cached by the alua checker. While this blurs the traditional separation between these components, for ALUA the path state and priority fundamentally derive from the same underlying data, making the coupling a natural fit.
This patch sets the alua checker as the auto-detected default for ALUA devices to demonstrate its benefits. We recognize that adopting a new default checker requires careful consideration; the auto-detection can be adjusted or removed based on review feedback.
Brian Bunker (1):
checkers: add alua path checker
libmultipath/Makefile | 5 + libmultipath/checkers.c | 1 + libmultipath/checkers.h | 1 + libmultipath/checkers/Makefile | 3 +- libmultipath/checkers/alua.c | 426 ++++++++++++++++++++++++ libmultipath/checkers/alua.h | 15 + libmultipath/prio.c | 1 + libmultipath/prio.h | 1 + libmultipath/prioritizers/Makefile | 1 + libmultipath/prioritizers/alua_cached.c | 225 +++++++++++++ libmultipath/prioritizers/alua_rtpg.c | 12 +- libmultipath/prioritizers/alua_rtpg.h | 1 + libmultipath/prioritizers/sysfs.c | 7 +- libmultipath/propsel.c | 25 +- libmultipath/structs.c | 2 + libmultipath/structs.h | 3 + 16 files changed, 721 insertions(+), 8 deletions(-) create mode 100644 libmultipath/checkers/alua.c create mode 100644 libmultipath/checkers/alua.h create mode 100644 libmultipath/prioritizers/alua_cached.c
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH 1/1 checkers: add alua path checker
Add an 'alua' path checker that uses RTPG to check path state, and an 'alua_cached' prioritizer that uses the checker's cached ALUA state.
The checker maps ALUA Asymmetric Access States to path states:
- AAS_OPTIMIZED, AAS_NON_OPTIMIZED -> PATH_UP
- AAS_STANDBY -> PATH_GHOST
- AAS_TRANSITIONING -> PATH_PENDING
- AAS_UNAVAILABLE, AAS_OFFLINE -> PATH_DOWN
The alua_cached prioritizer first checks for cached state from the alua checker, then falls back to sysfs, then to issuing RTPG directly. This maintains backward compatibility with other checker configurations.
When detect_checker is enabled, the alua checker is auto-selected for devices with TPGS support. When the alua checker is in use, detect_prio auto-selects the alua_cached prioritizer.
--- libmultipath/Makefile | 5 + libmultipath/checkers.c | 1 + libmultipath/checkers.h | 1 + libmultipath/checkers/Makefile | 3 +- libmultipath/checkers/alua.c | 426 ++++++++++++++++++++++++ libmultipath/checkers/alua.h | 15 + libmultipath/prio.c | 1 + libmultipath/prio.h | 1 + libmultipath/prioritizers/Makefile | 1 + libmultipath/prioritizers/alua_cached.c | 224 +++++++++++++ libmultipath/prioritizers/alua_rtpg.c | 12 +- libmultipath/prioritizers/alua_rtpg.h | 1 + libmultipath/prioritizers/sysfs.c | 7 +- libmultipath/propsel.c | 20 +- libmultipath/structs.c | 2 + libmultipath/structs.h | 3 + 16 files changed, 716 insertions(+), 7 deletions(-) create mode 100644 libmultipath/checkers/alua.c create mode 100644 libmultipath/checkers/alua.h create mode 100644 libmultipath/prioritizers/alua_cached.c
libmultipath/Makefile +5 -0
@@ -38,6 +38,11 @@ nvme-lib.o: nvme-lib.c nvme-ioctl.c nvme-ioctl.h dict.o: dict.c $(Q)$(CC) $(CPPFLAGS) $(CFLAGS) -Wno-unused-parameter -c -o $@ $< +# checkers/alua.o needs to find headers in parent directory +checkers/alua.o: checkers/alua.c + @echo building $@ because of $? + $(Q)$(CC) $(CPPFLAGS) $(CFLAGS) -I. -c -o $@ $< + make_static = $(shell sed '/^static/!s/^\([a-z]\{1,\} \)/static \1/' <$1 >$2) nvme-ioctl.c: nvme/nvme-ioctl.c
libmultipath/checkers.c +1 -0
@@ -461,6 +461,7 @@ int init_checkers(void)EMC_CLARIION ,READSECTOR0 ,CCISS_TUR , + ALUA_RTPG, };unsigned int i;
libmultipath/checkers.h +1 -0
@@ -98,6 +98,7 @@ enum path_check_state {#define EMC_CLARIION "emc_clariion"#define READSECTOR0 "readsector0"#define CCISS_TUR "cciss_tur" +#define ALUA "alua"#define NONE "none"#define INVALID "invalid"
libmultipath/checkers/Makefile +2 -1
@@ -17,7 +17,8 @@ LIBS= \ libcheckdirectio.so \ libcheckemc_clariion.so \ libcheckhp_sw.so \ - libcheckrdac.so + libcheckrdac.so \ + libcheckalua.so all: $(LIBS)
libmultipath/checkers/alua.c new file +426 -0
426 lines not on this page · raw message
libmultipath/checkers/alua.h new file +15 -0
@@ -0,0 +1,15 @@ +/* +* ALUA Path Checker Header +* +* This file is released under the GPL. +*/ +#ifndef _ALUA_CHECKER_H +#define _ALUA_CHECKER_H + +/* ALUA checker caches state in struct path for prioritizer use: +* pp->alua_state - cached AAS value (-1 if not cached) +* pp->alua_timestamp - when the state was last updated +*/ + +#endif /* _ALUA_CHECKER_H */ +
libmultipath/prio.c +1 -0
@@ -29,6 +29,7 @@ int init_prio(void)#ifdef LOAD_ALL_SHARED_LIBS static const char *const all_prios[] = {PRIO_ALUA , +PRIO_ALUA_CACHED ,PRIO_CONST ,PRIO_DATACORE , PRIO_EMC,
libmultipath/prio.h +1 -0
@@ -17,6 +17,7 @@ struct path;* Known prioritizers for use in hwtable.c */ #define PRIO_ALUA "alua" +#define PRIO_ALUA_CACHED "alua_cached"#define PRIO_CONST "const"#define PRIO_DATACORE "datacore"#define PRIO_EMC "emc"
libmultipath/prioritizers/Makefile +1 -0
@@ -13,6 +13,7 @@ LIBDEPS = -lmultipath -lmpathutil -lm -lpthread -lrt # If you add or remove a prioritizer also update multipath/multipath.conf.5 LIBS = \ libprioalua.so \ + libprioalua_cached.so \ libprioconst.so \ libpriodatacore.so \ libprioemc.so \
libmultipath/prioritizers/alua_cached.c new file +224 -0
224 lines not on this page · raw message
libmultipath/prioritizers/alua_rtpg.c +11 -1
@@ -75,6 +75,7 @@ enum scsi_disposition {SCSI_GOOD =0 ,SCSI_ERROR ,SCSI_RETRY , + SCSI_DISCONNECTED, }; static int
@@ -124,6 +125,12 @@ scsi_error(struct sg_io_hdr *hdr, int opcode)PRINT_DEBUG ("alua: SCSI error for command %02x: status %02x, sense %02x/%02x/%02x" , opcode, hdr->status, sense_key, asc, ascq); +/* Check for disconnected LUN (unmapped at target) */ +if (sense_key ==0x5 && asc ==0x25 && ascq ==0x00 ) { +/* ILLEGAL REQUEST / LOGICAL UNIT NOT SUPPORTED */ +return SCSI_DISCONNECTED ; + } +if (sense_key ==UNIT_ATTENTION || sense_key ==NOT_READY )return SCSI_RETRY ;else
@@ -325,7 +332,10 @@ retry: } rc =scsi_error (&hdr,OPERATION_CODE_RTPG ); -if (rc ==SCSI_ERROR ) { +if (rc ==SCSI_DISCONNECTED ) { +PRINT_DEBUG ("do_rtpg: LUN disconnected (unmapped at target)!" ); +return -RTPG_LUN_DISCONNECTED ; + }else if (rc ==SCSI_ERROR ) {PRINT_DEBUG ("do_rtpg: SCSI error!" );return -RTPG_RTPG_FAILED ; }else if (rc ==SCSI_RETRY ) {
libmultipath/prioritizers/alua_rtpg.h +1 -0
@@ -21,6 +21,7 @@#define RTPG_NO_TPG_IDENTIFIER 2#define RTPG_RTPG_FAILED 3#define RTPG_TPG_NOT_FOUND 4 +#define RTPG_LUN_DISCONNECTED 5int get_target_port_group_support (const struct path *pp);int get_target_port_group (const struct path *pp);
libmultipath/prioritizers/sysfs.c +5 -2
@@ -10,6 +10,9 @@#include "discovery.h" #include "prio.h" +/* Maximum length of sysfs access_state string */ +#define ALUA_ACCESS_STATE_SIZE 32 + static const struct { unsigned char value; char *name;
@@ -39,11 +42,11 @@ int get_exclusive_pref_arg(char *args)int getprio (struct path * pp,char *args) {int prio =0 , rc, i; -char buff[512 ]; +char buff[ALUA_ACCESS_STATE_SIZE ];int exclusive_pref; exclusive_pref =get_exclusive_pref_arg (args); - rc =sysfs_get_asymmetric_access_state (pp, buff,512 ); + rc =sysfs_get_asymmetric_access_state (pp, buff,sizeof (buff));if (rc <0 )return PRIO_UNDEF ; prio =0 ;
libmultipath/propsel.c +17 -3
@@ -262,6 +262,7 @@ verify_alua_prio(struct multipath *mp)if (!prio_selected (&pp->prio))continue ;if (strncmp (name,PRIO_ALUA ,PRIO_NAME_LEN ) && +strncmp (name,PRIO_ALUA_CACHED ,PRIO_NAME_LEN ) &&strncmp (name,PRIO_SYSFS ,PRIO_NAME_LEN ))return false; assume_alua = true;
@@ -690,7 +691,12 @@ int select_checker(struct config *conf, struct path *pp) } (void )path_get_tpgs (pp);if (pp->tpgs !=TPGS_NONE && pp->tpgs !=TPGS_UNDEF ) { - ckr_name =TUR ; +/* +* For ALUA devices (any TPGS support), use alua +* checker which combines path checking with ALUA state +* caching for optimal performance. +*/ + ckr_name =ALUA ; goto out; } }
@@ -767,8 +773,15 @@ void detect_prio(struct path *pp)if ((tpgs ==TPGS_EXPLICIT || !check_rdac (pp)) &&sysfs_get_asymmetric_access_state (pp, buff,512 ) >=0 ) default_prio =PRIO_SYSFS ; -else +else { default_prio =PRIO_ALUA ; +/* If using alua checker, use cached prioritizer */ +if (checker_selected (&pp->checker)) { +const char *chk_name =checker_name (&pp->checker); +if (chk_name &&strcmp (chk_name,ALUA ) ==0 ) + default_prio =PRIO_ALUA_CACHED ; + } + }break ;default :return ;
@@ -831,7 +844,8 @@ out:/* * fetch tpgs mode for alua, if its not already obtained */ -if (!strncmp (prio_name (p),PRIO_ALUA ,PRIO_NAME_LEN )) { +if (!strncmp (prio_name (p),PRIO_ALUA ,PRIO_NAME_LEN ) || + !strncmp (prio_name (p),PRIO_ALUA_CACHED ,PRIO_NAME_LEN )) {int tpgs =path_get_tpgs (pp);if (tpgs ==TPGS_NONE ) {
libmultipath/structs.c +2 -0
@@ -126,6 +126,8 @@ alloc_path (void) pp->fd =-1 ; pp->tpgs =TPGS_UNDEF ; pp->tpg_id =GROUP_ID_UNDEF ; + pp->alua_state =-1 ; + pp->alua_timestamp =0 ; pp->priority =PRIO_UNDEF ; pp->checkint =CHECKINT_UNDEF ;checker_clear (&pp->checker);
libmultipath/structs.h +3 -0
@@ -444,6 +444,9 @@ struct path {vector hwe;struct gen_path generic_path;int tpg_id; +/* Cached ALUA state from checker for prioritizer use */ +int alua_state;/* AAS value, -1 if not cached */ +time_t alua_timestamp;/* When alua_state was last updated */ enum ioctl_info_states ioctl_info; };
Signature · 1 line
2.50.1 (Apple Git-155)
Hi Brian,
Quoted 15 lines · Brian Bunker
Add an 'alua' path checker that uses RTPG to check path state, and an '
alua_cached' prioritizer that uses the checker's cached ALUA state.The checker maps ALUA Asymmetric Access States to path states:
-AAS_OPTIMIZED,AAS_NON_OPTIMIZED->PATH_UP
-AAS_STANDBY->PATH_GHOST
-AAS_TRANSITIONING->PATH_PENDING
-AAS_UNAVAILABLE,AAS_OFFLINE->PATH_DOWNThe
alua_cachedprioritizer first checks for cached state from the alua
checker, then falls back to sysfs, then to issuing RTPG directly. This
maintains backward compatibility with other checker configurations.
Quoted 6 lines
When
detect_checkeris enabled, the alua checker is auto-selected for devices with TPGS support. When the alua checker is in use,detect_prio
auto-selects thealua_cachedprioritizer.Signed-off-by Brian Bunker <[email protected]>
Thanks! Just a few remarks for now.
Martin
Quoted 87 lines
---
libmultipath/Makefile | 5 + libmultipath/checkers.c | 1 + libmultipath/checkers.h | 1 + libmultipath/checkers/Makefile | 3 +- libmultipath/checkers/alua.c | 426
++++++++++++++++++++++++
libmultipath/checkers/alua.h | 15 + libmultipath/prio.c | 1 + libmultipath/prio.h | 1 + libmultipath/prioritizers/Makefile | 1 +libmultipath/prioritizers/alua_cached.c| 224 +++++++++++++ libmultipath/prioritizers/alua_rtpg.c | 12 +- libmultipath/prioritizers/alua_rtpg.h | 1 + libmultipath/prioritizers/sysfs.c | 7 +-libmultipath/propsel.c | 20 +- libmultipath/structs.c | 2 + libmultipath/structs.h | 3 + 16 files changed, 716 insertions(+), 7 deletions(-)create mode 100644
libmultipath/checkers/alua.ccreate mode 100644libmultipath/checkers/alua.hcreate mode 100644libmultipath/prioritizers/alua_cached.clibmultipath/Makefile +0
-0@@ -38,6 +38,11 @@ nvme-lib.o: nvme-lib.c nvme-ioctl.c nvme-ioctl.hdict.o:
dict.c
$(Q)$(CC) $(CPPFLAGS) $(CFLAGS) -Wno-unused-parameter -c -o $@ $<libmultipath/Makefile +5
-0+# checkers/alua.o needs to find headers in parent directory +checkers/alua.o: checkers/alua.c + @echo building $@ because of $? + $(Q)$(CC) $(CPPFLAGS) $(CFLAGS) -I. -c -o $@ $< +make_static = $(shell sed '/^static/!s/^\([a-z]\{1,\} \)/static \1/' <$1 >$2)
nvme-ioctl.c:nvme/nvme-ioctl.clibmultipath/checkers.c +1
-0@@ -461,6 +461,7 @@ int init_checkers(void) EMC_CLARIION, READSECTOR0, CCISS_TUR, +ALUA_RTPG , }; unsigned int i;libmultipath/checkers.h +1
-0@@ -98,6 +98,7 @@ enum path_check_state { #define EMC_CLARIION "emc_clariion" #define READSECTOR0 "readsector0" #define CCISS_TUR "cciss_tur" +#define ALUA "alua" #define NONE "none" #define INVALID "invalid"libmultipath/checkers/Makefile +2
-1@@ -17,7 +17,8 @@ LIBS= \ libcheckdirectio.so \ libcheckemc_clariion.so \ libcheckhp_sw.so \ - libcheckrdac.so + libcheckrdac.so \ + libcheckalua.so
all: $(LIBS)libmultipath/checkers/alua.c new file +0
-0
libmultipath/checkers/alua.c line 5
Quoted 5 lines
libmultipath/checkers/alua.c new file +5
-0@@ -0,0 +1,426 @@ +/* +* ALUA Path Checker +* +* Copyright (c) 2024 +* This file is released under the GPL.
Please add a proper license header.
Quoted 63 lines
libmultipath/checkers/alua.c new file +63
-0+*/ +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <sys/sysmacros.h> +#include <unistd.h> +#include <fcntl.h> +#include <errno.h> +#include <pthread.h> +#include <urcu.h> +#include <urcu/uatomic.h> + +#include "checkers.h" +#include "debug.h" +#include "structs.h" +#include "prio.h" +#include "util.h" +#include "time-util.h" +#include "../prioritizers/alua.h" +#include "../prioritizers/alua_rtpg.h" +#include "alua.h" + +#define MAX_NR_TIMEOUTS 1 + +enum { +MSG_ALUA_RTPG_NOT_SUPPORTED =CHECKER_FIRST_MSGID , +MSG_ALUA_RTPG_TRANSITIONING , +MSG_ALUA_RUNNING , +MSG_ALUA_TIMEOUT , +}; + +#define IDX_ (x) (MSG_ ## x - CHECKER_FIRST_MSGID) +const char *libcheck_msgtable[] = { + [IDX_ (ALUA_RTPG_NOT_SUPPORTED )] =" ALUA not supported" , + [IDX_ (ALUA_RTPG_TRANSITIONING )] =" path is transitioning" , + [IDX_ (ALUA_RUNNING )] =" still running" , + [IDX_ (ALUA_TIMEOUT )] =" timed out" , +NULL , +}; + +struct alua_checker_context { +dev_t devt; +int state; +int running;/* uatomic access only */ +int fd; +unsigned int timeout; +time_t time; +pthread_t thread; +pthread_mutex_t lock; +pthread_cond_t active; +int holders;/* uatomic access only */ +int msgid; +struct checker_context ctx; +unsigned int nr_timeouts; +bool checked_state; +/* ALUA-specific cached data for prioritizer */ + int aas; /* Asymmetric Access State */ + int tpg; /* Target Port Group */ + time_t timestamp; /* When this data was collected */ +/* Data needed for RTPG in thread */ + struct path *pp; /* Path pointer - only valid during check
Quoted 2 lines
*/
+};
Instead of duplicating the threading implementation of the TUR checker here we should abstract it out and use the common framework by both checkers (and others, too, maybe). This is a thing I've wanted to do for a long time. Maybe it's time to start doing it now.
Quoted 393 lines
libmultipath/checkers/alua.c new file +105
-0+ +int libcheck_init (struct checker *c) +{ +struct alua_checker_context *ct; +struct stat sb; + + ct =malloc (sizeof (struct alua_checker_context )); +if (!ct) +return 1 ; +memset (ct,0 ,sizeof (struct alua_checker_context )); + + ct->state =PATH_UNCHECKED ; + ct->fd =-1 ; + ct->aas =-1 ; + ct->tpg =-1 ; +uatomic_set (&ct->holders,1 ); +pthread_cond_init_mono (&ct->active); +pthread_mutex_init (&ct->lock,NULL ); +if (fstat (c->fd, &sb) ==0 ) + ct->devt = sb.st_rdev; + ct->ctx.cls = c->cls; + c->context = ct; + +return 0 ; +} + +static void cleanup_context (struct alua_checker_context *ct) +{ +pthread_mutex_destroy (&ct->lock); +pthread_cond_destroy (&ct->active); +free (ct); +} + +void libcheck_free (struct checker *c) +{ +if (c->context) { +struct alua_checker_context *ct = c->context; +int holders; +int running; + + running =uatomic_xchg (&ct->running,0 ); +if (running) +pthread_cancel (ct->thread); + ct->thread =0 ; + holders =uatomic_sub_return (&ct->holders,1 ); +if (!holders) +cleanup_context (ct); + c->context =NULL ; + } +return ; +} + +/* +* Map ALUA Asymmetric Access State to path state +*/ +static int alua_state_to_path_state (int aas,short *msgid) +{ +switch (aas &0x0f ) { +case AAS_OPTIMIZED : +case AAS_NON_OPTIMIZED : + *msgid =CHECKER_MSGID_UP ; +return PATH_UP ; +case AAS_STANDBY : + *msgid =CHECKER_MSGID_GHOST ; +return PATH_GHOST ; +case AAS_TRANSITIONING : + *msgid =MSG_ALUA_RTPG_TRANSITIONING ; +return PATH_PENDING ; +case AAS_UNAVAILABLE : +case AAS_OFFLINE : +default : + *msgid =CHECKER_MSGID_DOWN ; +return PATH_DOWN ; + } +} + +/* +* Main ALUA check function - uses alua_rtpg.c library. +* This is called either in sync mode or from the async thread. +*/ +static int +alua_check (struct path *pp,int *aas_out,short *msgid) +{ +int aas; + +if (pp->tpg_id ==GROUP_ID_UNDEF ) { + *msgid =CHECKER_MSGID_DOWN ; +return PATH_DOWN ; + } + + aas =get_asymmetric_access_state (pp, pp->tpg_id); +if (aas <0 ) { +if (aas == -RTPG_LUN_DISCONNECTED ) { + *msgid =CHECKER_MSGID_DISCONNECTED ; +return PATH_DISCONNECTED ; + } + *msgid =CHECKER_MSGID_DOWN ; +return PATH_DOWN ; + } + + *aas_out = aas; +return alua_state_to_path_state (aas, msgid); +} + +#define alua_thread_cleanup_push (ct)
pthread_cleanup_push(cleanup_func, ct)libmultipath/checkers/alua.c new file +24
-0+#define alua_thread_cleanup_pop (ct) pthread_cleanup_pop(1) + +static void cleanup_func (void *data) +{ +int holders; +struct alua_checker_context *ct = data; + + holders =uatomic_sub_return (&ct->holders,1 ); +if (!holders) +cleanup_context (ct); +} + +void *libcheck_thread (struct checker_context *ctx) +{ +struct alua_checker_context *ct = + container_of(ctx, struct alua_checker_context, ctx); +int state, running; +short msgid; +int aas; + +/* This thread can be canceled, so setup clean up */ +alua_thread_cleanup_push (ct); + + condlog(4, "%d:%d : alua checker starting up", major(ct-Quoted 1 line
devt),
libmultipath/checkers/alua.c new file +19
-0+ minor(ct->devt)); + + state =alua_check (ct->pp, &aas, &msgid); +pthread_testcancel (); + +/* ALUA checker done */ +pthread_mutex_lock (&ct->lock); + ct->state = state; + ct->msgid = msgid; + ct->aas = aas; + ct->tpg = ct->pp->tpg_id; + ct->timestamp =time (NULL ); +/* Update path-level cache for prioritizer use */ + ct->pp->alua_state = aas; + ct->pp->alua_timestamp = ct->timestamp; +pthread_cond_signal (&ct->active); +pthread_mutex_unlock (&ct->lock); + + condlog(4, "%d:%d : alua checker finished, state %s",
major(ct->devt),libmultipath/checkers/alua.c new file +36
-0+ minor(ct->devt), checker_state_name(state)); + + running =uatomic_xchg (&ct->running,0 ); +if (!running) +pause (); + +alua_thread_cleanup_pop (ct); + +return ((void *)0 ); +} + +static void alua_set_async_timeout (struct checker *c) +{ +struct alua_checker_context *ct = c->context; +struct timespec now; + +get_monotonic_time (&now); + ct->time = now.tv_sec + c->timeout; +} + +static int alua_check_async_timeout (struct checker *c) +{ +struct alua_checker_context *ct = c->context; +struct timespec now; + +get_monotonic_time (&now); +return (now.tv_sec > ct->time); +} + +static int check_pending(struct checker *c) +{ + struct alua_checker_context *ct = c->context; + int alua_status = PATH_PENDING; + + pthread_mutex_lock(&ct->lock); + if (ct->state != PATH_PENDING || ct->msgid !=
MSG_ALUA_RUNNING) {libmultipath/checkers/alua.c new file +5
-0+ alua_status = ct->state; + c->msgid = ct->msgid; + } +pthread_mutex_unlock (&ct->lock); + if (alua_status == PATH_PENDING && c->msgid ==
MSG_ALUA_RUNNING) {libmultipath/checkers/alua.c new file +17
-0+condlog (4 ,"%d:%d : alua checker still running" , +major (ct->devt),minor (ct->devt)); + }else { +int running =uatomic_xchg (&ct->running,0 ); +if (running) +pthread_cancel (ct->thread); + ct->thread =0 ; + } + + ct->checked_state = true; +return alua_status; +} + +bool libcheck_need_wait (struct checker *c) +{ +struct alua_checker_context *ct = c->context; + return (ct && ct->thread && uatomic_read(&ct->running) != 0&&
libmultipath/checkers/alua.c new file +8
-0+ !ct->checked_state); +} + +int libcheck_pending (struct checker *c) +{ +struct alua_checker_context *ct = c->context; + +/* The if path checker isn't running, just return the exiting value. */
libmultipath/checkers/alua.c new file +36
-0+if (!ct || !ct->thread) +return c->path_state; + +return check_pending (c); +} + +int libcheck_check (struct checker *c) +{ +struct alua_checker_context *ct = c->context; + struct path *pp = container_of(c, struct path, checker); +pthread_attr_t attr; +int alua_status =PATH_PENDING , r; +int aas; + +if (!ct) +return PATH_UNCHECKED ; + +if (checker_is_sync (c)) { + alua_status =alua_check (pp, &aas, &c->msgid); + ct->tpg = pp->tpg_id; + ct->aas = aas; + ct->timestamp =time (NULL ); +/* Update path-level cache for prioritizer use */ + pp->alua_state = aas; + pp->alua_timestamp = ct->timestamp; +return alua_status; + } + +/* Async mode */ +if (ct->thread) { + ct->checked_state = true; +if (alua_check_async_timeout (c)) { +int running =uatomic_xchg (&ct->running,0 ); +if (running) { +pthread_cancel (ct->thread); + condlog(3, "%d:%d : alua checkertimeout",
libmultipath/checkers/alua.c new file +1
-0+ major(ct->devt), minor(ct-Quoted 1 line
devt));
libmultipath/checkers/alua.c new file +12
-0+ c->msgid =MSG_ALUA_TIMEOUT ; + alua_status =PATH_TIMEOUT ; + }else { +pthread_mutex_lock (&ct->lock); + alua_status = ct->state; + c->msgid = ct->msgid; +pthread_mutex_unlock (&ct->lock); + } + ct->thread =0 ; + }else if (uatomic_read (&ct->running) !=0 ) { +/* +* ct->running may be stale - the thread sets
libmultipath/checkers/alua.c new file +1
-0+* msgid/state under lock, then clears running
libmultipath/checkers/alua.c new file +1
-0+* after releasing it. Check msgid under lock
libmultipath/checkers/alua.c new file +1
-0+* to get the truth. Only MSG_ALUA_RUNNING means
libmultipath/checkers/alua.c new file +20
-0+* no result yet. +*/ +pthread_mutex_lock (&ct->lock); +if (ct->msgid !=MSG_ALUA_RUNNING ) { + alua_status = ct->state; + c->msgid = ct->msgid; + ct->thread =0 ; + } +pthread_mutex_unlock (&ct->lock); + }else { +/* ALUA checker done */ + ct->thread =0 ; +pthread_mutex_lock (&ct->lock); + alua_status = ct->state; + c->msgid = ct->msgid; +pthread_mutex_unlock (&ct->lock); + } + }else { +if (uatomic_read (&ct->holders) >1 ) { +/* The thread has been cancelled but hasn't quit. */
libmultipath/checkers/alua.c new file +2
-0+if (ct->nr_timeouts <=MAX_NR_TIMEOUTS ) { + if (ct->nr_timeouts ==
MAX_NR_TIMEOUTS) {libmultipath/checkers/alua.c new file +1
-0+ condlog(2, "%d:%d : waitingfor stalled alua thread to finish",
libmultipath/checkers/alua.c new file +1
-0+ major(ct->devt),
minor(ct->devt));libmultipath/checkers/alua.c new file +8
-0+ } + ct->nr_timeouts++; + c->msgid =MSG_ALUA_TIMEOUT ; +return PATH_TIMEOUT ; + } + ct->nr_timeouts++; +/* +* Start a new thread while the old one is stalled.
libmultipath/checkers/alua.c new file +1
-0+* We have to prevent it from interfering with the new
libmultipath/checkers/alua.c new file +1
-0+* thread. We create a new context and leave the old
libmultipath/checkers/alua.c new file +1
-0+* one with the stale thread, hoping it will clean up
libmultipath/checkers/alua.c new file +3
-0+* eventually. +*/ + condlog(3, "%d:%d : alua thread notresponding",
libmultipath/checkers/alua.c new file +7
-0+ major(ct->devt), minor(ct->devt)); + +if (libcheck_init (c) !=0 ) { + c->msgid =CHECKER_MSGID_DOWN ; +return PATH_UNCHECKED ; + } + ((struct alua_checker_context *)c->context)-Quoted 1 line
nr_timeouts=ct->nr_timeouts;libmultipath/checkers/alua.c new file +5
-0+ +if (!uatomic_sub_return (&ct->holders,1 )) { +/* It did terminate, eventually */ +cleanup_context (ct); + ((struct alua_checker_context *)c-Quoted 1 line
context)->nr_timeouts = 0;
libmultipath/checkers/alua.c new file +20
-0+ } + + ct = c->context; + }else + ct->nr_timeouts =0 ; + +/* Start new ALUA checker */ +pthread_mutex_lock (&ct->lock); + alua_status = ct->state =PATH_PENDING ; + c->msgid = ct->msgid =MSG_ALUA_RUNNING ; +pthread_mutex_unlock (&ct->lock); + ct->fd = c->fd; + ct->timeout = c->timeout; + ct->pp = pp; + ct->checked_state = false; +uatomic_add (&ct->holders,1 ); +uatomic_set (&ct->running,1 ); +alua_set_async_timeout (c); +setup_thread_attr (&attr,32 *1024 ,1 ); + r = start_checker_thread(&ct->thread, &attr, &ct-Quoted 1 line
ctx);
libmultipath/checkers/alua.c new file +6
-0+pthread_attr_destroy (&attr); +if (r) { +uatomic_sub (&ct->holders,1 ); +uatomic_set (&ct->running,0 ); + ct->thread =0 ; + condlog(3, "%d:%d : failed to start aluathread, using"
libmultipath/checkers/alua.c new file +1
-0+ " sync mode", major(ct->devt),
minor(ct->devt));libmultipath/checkers/alua.c new file +1
-0+ alua_status = alua_check(pp, &aas, &c-Quoted 1 line
msgid);
libmultipath/checkers/alua.c new file +4
-0+ ct->tpg = pp->tpg_id; + ct->aas = aas; + ct->timestamp =time (NULL ); +/* Update path-level cache for prioritizer use */
libmultipath/checkers/alua.c new file +9
-0+ pp->alua_state = aas; + pp->alua_timestamp = ct->timestamp; +return alua_status; + } + } + +return alua_status; +} +libmultipath/checkers/alua.h new file +0
-0
libmultipath/checkers/alua.h line 4
Quoted 4 lines
libmultipath/checkers/alua.h new file +4
-0@@ -0,0 +1,15 @@ +/* +* ALUA Path Checker Header +* +* This file is released under the GPL.
Please add a proper license header.
libmultipath/checkers/alua.h line 15
Quoted 11 lines
libmultipath/checkers/alua.h new file +11
-0+*/ +#ifndef _ALUA_CHECKER_H +#define _ALUA_CHECKER_H + +/* ALUA checker caches state in struct path for prioritizer use: +* pp->alua_state - cached AAS value (-1 if not cached) +* pp->alua_timestamp - when the state was last updated +*/ + +#endif /* _ALUA_CHECKER_H */ +
Only comments in this file?
Quoted 36 lines
libmultipath/prio.c +1
-0@@ -29,6 +29,7 @@ int init_prio(void) #ifdef LOAD_ALL_SHARED_LIBS static const char *const all_prios[] = { PRIO_ALUA, +PRIO_ALUA_CACHED , PRIO_CONST, PRIO_DATACORE, PRIO_EMC,libmultipath/prio.h +1
-0@@ -17,6 +17,7 @@ struct path;* Known prioritizers for use in hwtable.c */ #define PRIO_ALUA "alua" +#define PRIO_ALUA_CACHED "alua_cached" #define PRIO_CONST "const" #define PRIO_DATACORE "datacore" #define PRIO_EMC "emc"libmultipath/prioritizers/Makefile +0
-0@@ -13,6 +13,7 @@ LIBDEPS = -lmultipath -lmpathutil -lm -lpthread -lrt
# If you add or remove a prioritizer also update
multipath/multipath.conf.5
LIBS = \
libprioalua.so \libmultipath/prioritizers/Makefile +1
-0+ libprioalua_cached.so \
Quoted 3 lines
libprioconst.so \
libpriodatacore.so \
libprioemc.so \
I think the below should go into a separate patch.
Quoted 7 lines
libmultipath/prioritizers/alua_cached.c new file +0
-0
libmultipath/prioritizers/alua_cached.c line 8
Quoted 8 lines
libmultipath/prioritizers/alua_cached.c new file +8
-0@@ -0,0 +1,224 @@ +/* +* (C) Copyright IBM Corp. 2004, 2005 All Rights Reserved. +* Modified 2024 to support cached ALUA state from checker +* +* ALUA prioritizer that can use cached state from alua checker +* to avoid duplicate RTPG commands. +* +* This file is released under the GPL.
Please add a proper license header.
Quoted 24 lines
libmultipath/prioritizers/alua_cached.c new file +24
-0+*/ +#include <stdio.h> +#include <string.h> + +#include <time.h> + +#include "debug.h" +#include "prio.h" +#include "structs.h" +#include "checkers.h" +#include "discovery.h" + +#include "alua.h" + +#define ALUA_PRIO_NOT_SUPPORTED 1 +#define ALUA_PRIO_RTPG_FAILED 2 +#define ALUA_PRIO_GETAAS_FAILED 3 +#define ALUA_PRIO_TPGS_FAILED 4 +#define ALUA_PRIO_NO_INFORMATION 5 +#define ALUA_PRIO_CACHED_STALE 6 + +/* Maximum age of cached ALUA state in seconds */ +#define ALUA_CACHE_MAX_AGE_SECS 5 +
libmultipath/prioritizers/alua_cached.c line 42
Quoted 10 lines
libmultipath/prioritizers/alua_cached.c new file +10
-0+static const char * aas_string[] = { + [AAS_OPTIMIZED ] ="active/optimized" , + [AAS_NON_OPTIMIZED ] ="active/non-optimized" , + [AAS_STANDBY ] ="standby" , + [AAS_UNAVAILABLE ] ="unavailable" , + [AAS_LBA_DEPENDENT ] ="logical block dependent" , + [AAS_RESERVED ] ="ARRAY BUG: invalid TPGs state!" , + [AAS_OFFLINE ] ="offline" , + [AAS_TRANSITIONING ] ="transitioning between states" , +};
We shouldn't duplicate the definition of these constants.
Quoted 378 lines
libmultipath/prioritizers/alua_cached.c new file +22
-0+ +static const char *aas_print_string (int rc) +{ + rc &=0x7f ; + +if (rc &0x70 ) +return aas_string[AAS_RESERVED ]; + rc &=0x0f ; +if (rc >AAS_RESERVED && rc <AAS_OFFLINE ) +return aas_string[AAS_RESERVED ]; +else +return aas_string[rc]; +} + +/* +* Try to get ALUA info with optimization: +* 1. If using alua checker: use its cached state (it just polled!) +* 2. Try sysfs (kernel maintains this, no I/O needed) +* 3. Fall back to issuing RTPG directly +* +* Rationale: The checker is our active poller. If we're using alua +* and its cache is fresh, we trust that data first. If the cache misses
+ * (stale or empty), we try sysfs before issuing RTPG. This provides a
+ * three-tier optimization: checker cache → sysfs → RTPG.
+ */libmultipath/prioritizers/alua_cached.c new file +14
-0+/* Maximum length of sysfs access_state string */ +#define ALUA_ACCESS_STATE_SIZE 32 + +int +get_alua_info_cached (struct path * pp,bool *used_cache) +{ +int rc; +int tpg; +bool diff_tpg; +char buff[ALUA_ACCESS_STATE_SIZE ]; + + *used_cache = false; + +/* First: if using alua checker, use its fresh data from path structure */
libmultipath/prioritizers/alua_cached.c new file +9
-0+ if (checker_selected(&pp->checker)) { + const char *chk_name = checker_name(&pp->checker); + + if (chk_name && strcmp(chk_name, "alua") == 0 && + pp->alua_state >= 0) { + time_t now = time(NULL); + time_t age = now - pp->alua_timestamp; + + if (pp->alua_timestamp > 0 && age <=
ALUA_CACHE_MAX_AGE_SECS) {libmultipath/prioritizers/alua_cached.c new file +1
-0+ condlog(4, "%s: using cached ALUAstate from checker (fresh poll)",
pp->dev);libmultipath/prioritizers/alua_cached.c new file +4
-0+ *used_cache = true; +return pp->alua_state; + } + condlog(4, "%s: cached ALUA state notavailable or stale, trying sysfs",
libmultipath/prioritizers/alua_cached.c new file +2
-0+ pp->dev); +/* Cache miss - try sysfs before issuing RTPG */
libmultipath/prioritizers/alua_cached.c new file +4
-0+ } + } + +/* Second: try sysfs (if cache missed or not using alua checker) */
libmultipath/prioritizers/alua_cached.c new file +1
-0+ rc = sysfs_get_asymmetric_access_state(pp, buff,sizeof(buff));
libmultipath/prioritizers/alua_cached.c new file +16
-0+if (rc >=0 ) { +/* Parse sysfs state string to ALUA state value */ +int aas =-1 ; +int priopath = rc;/* rc contains preferred bit */ + +if (!strncmp (buff,"active/optimized" ,16 )) + aas =AAS_OPTIMIZED ; +else if (!strncmp (buff,"active/non-optimized" ,20 )) + aas =AAS_NON_OPTIMIZED ; +else if (!strncmp (buff,"lba-dependent" ,13 )) + aas =AAS_LBA_DEPENDENT ; +else if (!strncmp (buff,"standby" ,7 )) + aas =AAS_STANDBY ; + +if (aas >=0 ) { + condlog(4, "%s: using sysfs ALUA state (noI/O)",
pp->dev);libmultipath/prioritizers/alua_cached.c new file +16
-0+ *used_cache = true; +return (priopath ?0x80 :0 ) | aas; + } + } + +/* Last resort: issue RTPG directly */ + tpg =get_target_port_group (pp); +if (tpg <0 ) { + rc =get_target_port_group_support (pp); +if (rc <0 ) +return -ALUA_PRIO_TPGS_FAILED ; +if (rc ==TPGS_NONE ) +return -ALUA_PRIO_NOT_SUPPORTED ; +return -ALUA_PRIO_RTPG_FAILED ; + } + diff_tpg = (pp->tpg_id != GROUP_ID_UNDEF && pp->tpg_id !=tpg);
libmultipath/prioritizers/alua_cached.c new file +2
-0+ pp->tpg_id = tpg; + condlog((diff_tpg) ? 2 : 4, "%s: reported target port groupis %i",
libmultipath/prioritizers/alua_cached.c new file +4
-0+ pp->dev, tpg); + rc =get_asymmetric_access_state (pp, tpg); +if (rc <0 ) { + condlog(2, "%s: get_asymmetric_access_state returned%d",
libmultipath/prioritizers/alua_cached.c new file +5
-0+ __func__, rc); +return -ALUA_PRIO_GETAAS_FAILED ; + } + + condlog(3, "%s: aas = %02x [%s]%s", pp->dev, rc,
aas_print_string(rc),libmultipath/prioritizers/alua_cached.c new file +36
-0+ (rc & 0x80) ? " [preferred]" : ""); +return rc; +} + +int get_exclusive_pref_arg (char *args) +{ +char *ptr; + +if (args ==NULL ) +return 0 ; + ptr =strstr (args,"exclusive_pref_bit" ); +if (!ptr) +return 0 ; +if (ptr[18 ] !='\0' && ptr[18 ] !=' ' && ptr[18 ] !='\t' ) +return 0 ; +if (ptr != args && ptr[-1 ] !=' ' && ptr[-1 ] !='\t' ) +return 0 ; +return 1 ; +} + +int getprio (struct path * pp,char * args) +{ +int rc; +int aas; +int priopath; +int exclusive_pref; +bool used_cache = false; + +if (pp->fd <0 ) +return -ALUA_PRIO_NO_INFORMATION ; + + exclusive_pref =get_exclusive_pref_arg (args); + rc =get_alua_info_cached (pp, &used_cache); + +if (used_cache) { + condlog(4, "%s: priority calculated from cached ALUAstate (no RTPG issued)",
libmultipath/prioritizers/alua_cached.c new file +23
-0+ pp->dev); + } + +if (rc >=0 ) { + aas = (rc &0x0f ); + priopath = (rc &0x80 ); +switch (aas) { +case AAS_OPTIMIZED : + rc =50 ; +break ; +case AAS_NON_OPTIMIZED : + rc =10 ; +break ; +case AAS_LBA_DEPENDENT : + rc =5 ; +break ; +case AAS_STANDBY : + rc =1 ; +break ; +default : + rc =0 ; + } + if (priopath && (aas != AAS_OPTIMIZED ||
exclusive_pref))libmultipath/prioritizers/alua_cached.c new file +5
-0+ rc +=80 ; + }else { +switch (-rc) { +case ALUA_PRIO_NOT_SUPPORTED : + condlog(0, "%s: alua not supported", pp-Quoted 1 line
dev);
libmultipath/prioritizers/alua_cached.c new file +3
-0+break ; +case ALUA_PRIO_RTPG_FAILED : + condlog(0, "%s: couldn't get target portgroup",
pp->dev);libmultipath/prioritizers/alua_cached.c new file +3
-0+break ; +case ALUA_PRIO_GETAAS_FAILED : + condlog(0, "%s: couldn't get asymmetricaccess state",
pp->dev);libmultipath/prioritizers/alua_cached.c new file +3
-0+break ; +case ALUA_PRIO_TPGS_FAILED : + condlog(3, "%s: couldn't get supported aluastates",
pp->dev);libmultipath/prioritizers/alua_cached.c new file +6
-0+break ; + } + } +return rc; +} +libmultipath/prioritizers/alua_rtpg.c +1
-0@@ -75,6 +75,7 @@ enum scsi_disposition { SCSI_GOOD = 0, SCSI_ERROR, SCSI_RETRY, + SCSI_DISCONNECTED, };
static intlibmultipath/prioritizers/alua_rtpg.c +0
-0@@ -124,6 +125,12 @@ scsi_error(struct sg_io_hdr *hdr, int opcode) PRINT_DEBUG("alua: SCSI error for command %02x: status %02x, sense %02x/%02x/%02x", opcode, hdr->status, sense_key, asc, ascq);libmultipath/prioritizers/alua_rtpg.c +6
-0+/* Check for disconnected LUN (unmapped at target) */ +if (sense_key ==0x5 && asc ==0x25 && ascq ==0x00 ) { +/* ILLEGAL REQUEST / LOGICAL UNIT NOT SUPPORTED */ +return SCSI_DISCONNECTED ; + } +if (
sense_key==UNIT_ATTENTION||sense_key==NOT_READY) returnSCSI_RETRY;
elselibmultipath/prioritizers/alua_rtpg.c +0
-0@@ -325,7 +332,10 @@ retry: }
rc =scsi_error(&hdr, OPERATION_CODE_RTPG);libmultipath/prioritizers/alua_rtpg.c +2
-1-if (rc ==SCSI_ERROR ) { +if (rc ==SCSI_DISCONNECTED ) { + PRINT_DEBUG("do_rtpg: LUN disconnected (unmapped attarget)!");
libmultipath/prioritizers/alua_rtpg.c +2
-0+return -RTPG_LUN_DISCONNECTED ; + }else if (rc ==SCSI_ERROR ) {
PRINT_DEBUG("do_rtpg: SCSI error!"); return -RTPG_RTPG_FAILED;
} else if (rc ==SCSI_RETRY) {libmultipath/prioritizers/alua_rtpg.h +1
-0@@ -21,6 +21,7 @@ #define RTPG_NO_TPG_IDENTIFIER 2 #define RTPG_RTPG_FAILED 3 #define RTPG_TPG_NOT_FOUND 4 +#define RTPG_LUN_DISCONNECTED 5
intget_target_port_group_support(const struct path *pp); intget_target_port_group(const struct path *pp);libmultipath/prioritizers/sysfs.c +0
-0@@ -10,6 +10,9 @@ #include "discovery.h" #include "prio.h"libmultipath/prioritizers/sysfs.c +3
-0+/* Maximum length of sysfs access_state string */ +#define ALUA_ACCESS_STATE_SIZE 32 +static const struct {
unsigned char value;
char *name;libmultipath/prioritizers/sysfs.c +1
-1@@ -39,11 +42,11 @@ int get_exclusive_pref_arg(char *args) int getprio (struct path * pp, char *args) { int prio = 0, rc, i; -char buff[512 ]; +char buff[ALUA_ACCESS_STATE_SIZE ]; int exclusive_pref;
exclusive_pref=get_exclusive_pref_arg(args);libmultipath/prioritizers/sysfs.c +1
-1- rc =sysfs_get_asymmetric_access_state (pp, buff,512 ); + rc = sysfs_get_asymmetric_access_state(pp, buff,sizeof(buff));
if (rc < 0)
returnPRIO_UNDEF;
prio = 0;libmultipath/propsel.c +0
-0@@ -262,6 +262,7 @@ verify_alua_prio(struct multipath *mp) if (!prio_selected(&pp->prio)) continue;if (
strncmp(name, PRIO_ALUA, PRIO_NAME_LEN)&&libmultipath/propsel.c +1
-0+ strncmp(name, PRIO_ALUA_CACHED, PRIO_NAME_LEN)&&
strncmp(name, PRIO_SYSFS, PRIO_NAME_LEN)) return false;
assume_alua= true;libmultipath/propsel.c +2
-1@@ -690,7 +691,12 @@ int select_checker(struct config *conf, struct path *pp) } (void)path_get_tpgs(pp); if (pp->tpgs != TPGS_NONE && pp->tpgs != TPGS_UNDEF) { - ckr_name =TUR ; +/* +* For ALUA devices (any TPGS support), use alua
libmultipath/propsel.c +1
-0+* checker which combines path checking with ALUA state
libmultipath/propsel.c +3
-0+* caching for optimal performance. +*/ + ckr_name =ALUA ;goto out;
}
}libmultipath/propsel.c +0
-0@@ -767,8 +773,15 @@ void detect_prio(struct path *pp)if ((tpgs ==
TPGS_EXPLICIT|| !check_rdac(pp)) &&sysfs_get_asymmetric_access_state(pp, buff, 512)Quoted 1 line
= 0)
default_prio=PRIO_SYSFS;libmultipath/propsel.c +1
-1-else +else {
default_prio=PRIO_ALUA;libmultipath/propsel.c +1
-0+/* If using alua checker, use cached prioritizer */
libmultipath/propsel.c +2
-0+if (checker_selected (&pp->checker)) { + const char *chk_name =
checker_name(&pp->checker);libmultipath/propsel.c +1
-0+ if (chk_name && strcmp(chk_name,ALUA) == 0)
libmultipath/propsel.c +1
-0+ default_prio =
PRIO_ALUA_CACHED;libmultipath/propsel.c +2
-0+ } + }break;
default:
return;libmultipath/propsel.c +2
-1@@ -831,7 +844,8 @@ out: /** fetch tpgs mode for alua, if its not already obtained */ -if (!strncmp (prio_name (p),PRIO_ALUA ,PRIO_NAME_LEN )) { +if (!strncmp (prio_name (p),PRIO_ALUA ,PRIO_NAME_LEN ) || + !strncmp(prio_name(p), PRIO_ALUA_CACHED, PRIO_NAME_LEN)) { int tpgs = path_get_tpgs(pp);
if (tpgs ==TPGS_NONE) {libmultipath/structs.c +2
-0@@ -126,6 +126,8 @@ alloc_path (void) pp->fd = -1; pp->tpgs = TPGS_UNDEF; pp->tpg_id = GROUP_ID_UNDEF; + pp->alua_state =-1 ; + pp->alua_timestamp =0 ; pp->priority = PRIO_UNDEF; pp->checkint = CHECKINT_UNDEF; checker_clear(&pp->checker);libmultipath/structs.h +2
-0@@ -444,6 +444,9 @@ struct path { vector hwe; struct gen_path generic_path; int tpg_id; +/* Cached ALUA state from checker for prioritizer use */ +int alua_state;/* AAS value, -1 if not cached */
libmultipath/structs.h +1
-0+time_t alua_timestamp;/* When alua_state was last updated */
enumioctl_info_statesioctl_info;
};
A message that is not in the archive (sent off-list or not yet received): CAHZQxy+V5YSiHnB5sp8A_jbN_n1OQ633KiDeFa+vXQrNafHzDA@mail.gmail.com
Quoted 59 lines · Brian Bunker
Quoted 57 lines · Martin Wilck
Hi Brian,
Quoted 19 lines · Brian Bunker
Add an 'alua' path checker that uses RTPG to check path state, and an
'alua_cached' prioritizer that uses the checker's cached ALUA state.The checker maps ALUA Asymmetric Access States to path states:
-AAS_OPTIMIZED,AAS_NON_OPTIMIZED->PATH_UP
-AAS_STANDBY->PATH_GHOST
-AAS_TRANSITIONING->PATH_PENDING
-AAS_UNAVAILABLE,AAS_OFFLINE->PATH_DOWNThe
alua_cachedprioritizer first checks for cached state from the
alua
checker, then falls back to sysfs, then to issuing RTPG directly. This
maintains backward compatibility with other checker configurations.Quoted 7 lines
When
detect_checkeris enabled, the alua checker is auto-selected for
devices with TPGS support. When the alua checker is in use,detect_prio
auto-selects thealua_cachedprioritizer.Signed-off-by Brian Bunker <[email protected]>Thanks! Just a few remarks for now.
Martin
Quoted 22 lines
---
libmultipath/Makefile | 5 + libmultipath/checkers.c | 1 + libmultipath/checkers.h | 1 + libmultipath/checkers/Makefile | 3 +- libmultipath/checkers/alua.c | 426
++++++++++++++++++++++++
libmultipath/checkers/alua.h | 15 + libmultipath/prio.c | 1 + libmultipath/prio.h | 1 + libmultipath/prioritizers/Makefile | 1 +libmultipath/prioritizers/alua_cached.c| 224 +++++++++++++ libmultipath/prioritizers/alua_rtpg.c | 12 +- libmultipath/prioritizers/alua_rtpg.h | 1 + libmultipath/prioritizers/sysfs.c | 7 +-libmultipath/propsel.c | 20 +- libmultipath/structs.c | 2 + libmultipath/structs.h | 3 + 16 files changed, 716 insertions(+), 7 deletions(-)create mode 100644
libmultipath/checkers/alua.ccreate mode 100644libmultipath/checkers/alua.hcreate mode 100644libmultipath/prioritizers/alua_cached.c
[...]
Quoted 7 lines
Quoted 7 lines
Instead of duplicating the threading implementation of the TUR checker
here we should abstract it out and use the common framework by both checkers (and others, too, maybe). This is a thing I've wanted to do
for a long time. Maybe it's time to start doing it now.
Quoted 6 lines
I considered that sort of re-work, but I was concerned that changes adding to the existing checker might muddy the waters. There are certainly
changes where helper functions could be extracted for both checkers to leverage them.
I didn't mean to say that _you_ have to do it. I see it as my job, actually. The question is whether you are willing to wait until I get to it...
[...]
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
libmultipath/prioritizers/Makefile +0
-1@@ -13,6 +13,7 @@ LIBDEPS = -lmultipath -lmpathutil -lm -lpthread -lrt
# If you add or remove a prioritizer also update
multipath/multipath.conf.5Quoted 9 lines
Quoted 6 lines
LIBS = \
libprioalua.so \
+ libprioalua_cached.so \
libprioconst.so \
libpriodatacore.so \
libprioemc.so \I think the below should go into a separate patch.
Quoted 3 lines
Can they be separated? The prioritizer's logic relies on the checker populating values for its decision.
First the checker, then the prioritizer?
Quoted 56 lines
Quoted 47 lines
Quoted 8 lines
libmultipath/prioritizers/alua_cached.c new file +0
-0libmultipath/prioritizers/alua_cached.c line 8
Quoted 8 lines
libmultipath/prioritizers/alua_cached.c new file +8
-0@@ -0,0 +1,224 @@ +/* +* (C) Copyright IBM Corp. 2004, 2005 All Rights Reserved. +* Modified 2024 to support cached ALUA state from checker +* +* ALUA prioritizer that can use cached state from alua checker +* to avoid duplicate RTPG commands. +* +* This file is released under the GPL. Please add a proper license header.
Quoted 28 lines
libmultipath/prioritizers/alua_cached.c new file +28
-0+*/ +#include <stdio.h> +#include <string.h> + +#include <time.h> + +#include "debug.h" +#include "prio.h" +#include "structs.h" +#include "checkers.h" +#include "discovery.h" + +#include "alua.h" + +#define ALUA_PRIO_NOT_SUPPORTED 1 +#define ALUA_PRIO_RTPG_FAILED 2 +#define ALUA_PRIO_GETAAS_FAILED 3 +#define ALUA_PRIO_TPGS_FAILED 4 +#define ALUA_PRIO_NO_INFORMATION 5 +#define ALUA_PRIO_CACHED_STALE 6 + +/* Maximum age of cached ALUA state in seconds */ +#define ALUA_CACHE_MAX_AGE_SECS 5 + +static const char * aas_string[] = { + [AAS_OPTIMIZED] = "active/optimized", + [AAS_NON_OPTIMIZED] = "active/non-optimized", + [AAS_STANDBY] = "standby",Quoted 8 lines
libmultipath/prioritizers/alua_cached.c line 42
Quoted 6 lines
libmultipath/prioritizers/alua_cached.c new file +6
-0+ [AAS_UNAVAILABLE] = "unavailable", + [AAS_LBA_DEPENDENT] = "logical block dependent", + [AAS_RESERVED] = "ARRAY BUG: invalid TPGs state!", + [AAS_OFFLINE] = "offline", + [AAS_TRANSITIONING] = "transitioning between states", +};We shouldn't duplicate the definition of these constants.
Quoted 4 lines
This was somewhat intentional though not efficient. I will do better with leveraging in the next patch.
OK.
Thanks
Martin
Quoted 110 lines · Martin Wilck
Hi Brian,
Quoted 15 lines · Brian Bunker
Add an 'alua' path checker that uses RTPG to check path state, and an '
alua_cached' prioritizer that uses the checker's cached ALUA state.The checker maps ALUA Asymmetric Access States to path states:
-AAS_OPTIMIZED,AAS_NON_OPTIMIZED->PATH_UP
-AAS_STANDBY->PATH_GHOST
-AAS_TRANSITIONING->PATH_PENDING
-AAS_UNAVAILABLE,AAS_OFFLINE->PATH_DOWNThe
alua_cachedprioritizer first checks for cached state from the alua
checker, then falls back to sysfs, then to issuing RTPG directly. This
maintains backward compatibility with other checker configurations.Quoted 6 lines
When
detect_checkeris enabled, the alua checker is auto-selected for devices with TPGS support. When the alua checker is in use,detect_prio
auto-selects thealua_cachedprioritizer.Signed-off-by Brian Bunker <[email protected]>Thanks! Just a few remarks for now.
Martin
Quoted 80 lines
--- libmultipath/Makefile | 5 + libmultipath/checkers.c | 1 + libmultipath/checkers.h | 1 + libmultipath/checkers/Makefile | 3 +- libmultipath/checkers/alua.c | 426 ++++++++++++++++++++++++ libmultipath/checkers/alua.h | 15 + libmultipath/prio.c | 1 + libmultipath/prio.h | 1 + libmultipath/prioritizers/Makefile | 1 + libmultipath/prioritizers/alua_cached.c | 224 +++++++++++++ libmultipath/prioritizers/alua_rtpg.c | 12 +- libmultipath/prioritizers/alua_rtpg.h | 1 + libmultipath/prioritizers/sysfs.c | 7 +- libmultipath/propsel.c | 20 +- libmultipath/structs.c | 2 + libmultipath/structs.h | 3 + 16 files changed, 716 insertions(+), 7 deletions(-) create mode 100644 libmultipath/checkers/alua.c create mode 100644 libmultipath/checkers/alua.h create mode 100644 libmultipath/prioritizers/alua_cached.clibmultipath/Makefile +0
-0@@ -38,6 +38,11 @@ nvme-lib.o: nvme-lib.c nvme-ioctl.c nvme-ioctl.h dict.o: dict.c $(Q)$(CC) $(CPPFLAGS) $(CFLAGS) -Wno-unused-parameter -c -o$@ $<
libmultipath/Makefile +5
-0+# checkers/alua.o needs to find headers in parent directory +checkers/alua.o: checkers/alua.c + @echo building $@ because of $? + $(Q)$(CC) $(CPPFLAGS) $(CFLAGS) -I. -c -o $@ $< + make_static = $(shell sed '/^static/!s/^\([a-z]\{1,\} \)/static \1/'<$1 >$2)
nvme-ioctl.c:nvme/nvme-ioctl.clibmultipath/checkers.c +1
-0@@ -461,6 +461,7 @@ int init_checkers(void)EMC_CLARIION ,READSECTOR0 ,CCISS_TUR , + ALUA_RTPG, };unsigned int i;libmultipath/checkers.h +1
-0@@ -98,6 +98,7 @@ enum path_check_state {#define EMC_CLARIION "emc_clariion"#define READSECTOR0 "readsector0"#define CCISS_TUR "cciss_tur" +#define ALUA "alua"#define NONE "none"#define INVALID "invalid"libmultipath/checkers/Makefile +2
-1@@ -17,7 +17,8 @@ LIBS= \ libcheckdirectio.so \ libcheckemc_clariion.so \ libcheckhp_sw.so \ - libcheckrdac.so + libcheckrdac.so \ + libcheckalua.so all: $(LIBS)
Quoted 14 lines
libmultipath/checkers/alua.c line 5
Quoted 12 lines
libmultipath/checkers/alua.c new file +5
-0@@ -0,0 +1,426 @@ +/* +* ALUA Path Checker +* +* Copyright (c) 2024 +* This file is released under the GPL. Please add a proper license header.
I will do that.
Quoted 63 lines
Quoted 63 lines
libmultipath/checkers/alua.c new file +62
-0+*/ +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <sys/sysmacros.h> +#include <unistd.h> +#include <fcntl.h> +#include <errno.h> +#include <pthread.h> +#include <urcu.h> +#include <urcu/uatomic.h> + +#include "checkers.h" +#include "debug.h" +#include "structs.h" +#include "prio.h" +#include "util.h" +#include "time-util.h" +#include "../prioritizers/alua.h" +#include "../prioritizers/alua_rtpg.h" +#include "alua.h" + +#define MAX_NR_TIMEOUTS 1 + +enum { +MSG_ALUA_RTPG_NOT_SUPPORTED =CHECKER_FIRST_MSGID , +MSG_ALUA_RTPG_TRANSITIONING , +MSG_ALUA_RUNNING , +MSG_ALUA_TIMEOUT , +}; + +#define IDX_ (x) (MSG_ ## x - CHECKER_FIRST_MSGID) +const char *libcheck_msgtable[] = { + [IDX_ (ALUA_RTPG_NOT_SUPPORTED )] =" ALUA not supported" , + [IDX_ (ALUA_RTPG_TRANSITIONING )] =" path is transitioning" , + [IDX_ (ALUA_RUNNING )] =" still running" , + [IDX_ (ALUA_TIMEOUT )] =" timed out" , +NULL , +}; + +struct alua_checker_context { +dev_t devt; +int state; +int running;/* uatomic access only */ +int fd; +unsigned int timeout; +time_t time; +pthread_t thread; +pthread_mutex_t lock; +pthread_cond_t active; +int holders;/* uatomic access only */ +int msgid; +struct checker_context ctx; +unsigned int nr_timeouts; +bool checked_state; +/* ALUA-specific cached data for prioritizer */ +int aas;/* Asymmetric Access State */ +int tpg;/* Target Port Group */ +time_t timestamp;/* When this data was collected */ +/* Data needed for RTPG in thread */
Quoted 8 lines
Quoted 3 lines
libmultipath/checkers/alua.c new file +1
-0+struct path *pp;/* Path pointer - only valid during check */
+};Instead of duplicating the threading implementation of the TUR checker here we should abstract it out and use the common framework by both checkers (and others, too, maybe). This is a thing I've wanted to do for a long time. Maybe it's time to start doing it now.
I considered that sort of re-work, but I was concerned that changes adding to the existing checker might muddy the waters. There are certainly changes where helper functions could be extracted for both checkers to leverage them.
Quoted 404 lines
Quoted 397 lines
libmultipath/checkers/alua.c new file +105
-0+ +int libcheck_init (struct checker *c) +{ +struct alua_checker_context *ct; +struct stat sb; + + ct =malloc (sizeof (struct alua_checker_context )); +if (!ct) +return 1 ; +memset (ct,0 ,sizeof (struct alua_checker_context )); + + ct->state =PATH_UNCHECKED ; + ct->fd =-1 ; + ct->aas =-1 ; + ct->tpg =-1 ; +uatomic_set (&ct->holders,1 ); +pthread_cond_init_mono (&ct->active); +pthread_mutex_init (&ct->lock,NULL ); +if (fstat (c->fd, &sb) ==0 ) + ct->devt = sb.st_rdev; + ct->ctx.cls = c->cls; + c->context = ct; + +return 0 ; +} + +static void cleanup_context (struct alua_checker_context *ct) +{ +pthread_mutex_destroy (&ct->lock); +pthread_cond_destroy (&ct->active); +free (ct); +} + +void libcheck_free (struct checker *c) +{ +if (c->context) { +struct alua_checker_context *ct = c->context; +int holders; +int running; + + running =uatomic_xchg (&ct->running,0 ); +if (running) +pthread_cancel (ct->thread); + ct->thread =0 ; + holders =uatomic_sub_return (&ct->holders,1 ); +if (!holders) +cleanup_context (ct); + c->context =NULL ; + } +return ; +} + +/* +* Map ALUA Asymmetric Access State to path state +*/ +static int alua_state_to_path_state (int aas,short *msgid) +{ +switch (aas &0x0f ) { +case AAS_OPTIMIZED : +case AAS_NON_OPTIMIZED : + *msgid =CHECKER_MSGID_UP ; +return PATH_UP ; +case AAS_STANDBY : + *msgid =CHECKER_MSGID_GHOST ; +return PATH_GHOST ; +case AAS_TRANSITIONING : + *msgid =MSG_ALUA_RTPG_TRANSITIONING ; +return PATH_PENDING ; +case AAS_UNAVAILABLE : +case AAS_OFFLINE : +default : + *msgid =CHECKER_MSGID_DOWN ; +return PATH_DOWN ; + } +} + +/* +* Main ALUA check function - uses alua_rtpg.c library. +* This is called either in sync mode or from the async thread. +*/ +static int +alua_check (struct path *pp,int *aas_out,short *msgid) +{ +int aas; + +if (pp->tpg_id ==GROUP_ID_UNDEF ) { + *msgid =CHECKER_MSGID_DOWN ; +return PATH_DOWN ; + } + + aas =get_asymmetric_access_state (pp, pp->tpg_id); +if (aas <0 ) { +if (aas == -RTPG_LUN_DISCONNECTED ) { + *msgid =CHECKER_MSGID_DISCONNECTED ; +return PATH_DISCONNECTED ; + } + *msgid =CHECKER_MSGID_DOWN ; +return PATH_DOWN ; + } + + *aas_out = aas; +return alua_state_to_path_state (aas, msgid); +} + +#define alua_thread_cleanup_push (ct)
pthread_cleanup_push(cleanup_func, ct)libmultipath/checkers/alua.c new file +24
-0+#define alua_thread_cleanup_pop (ct) pthread_cleanup_pop(1) + +static void cleanup_func (void *data) +{ +int holders; +struct alua_checker_context *ct = data; + + holders =uatomic_sub_return (&ct->holders,1 ); +if (!holders) +cleanup_context (ct); +} + +void *libcheck_thread (struct checker_context *ctx) +{ +struct alua_checker_context *ct = + container_of(ctx, struct alua_checker_context, ctx); +int state, running; +short msgid; +int aas; + +/* This thread can be canceled, so setup clean up */ +alua_thread_cleanup_push (ct); + + condlog(4, "%d:%d : alua checker starting up", major(ct-Quoted 1 line
devt),
libmultipath/checkers/alua.c new file +19
-0+ minor(ct->devt)); + + state =alua_check (ct->pp, &aas, &msgid); +pthread_testcancel (); + +/* ALUA checker done */ +pthread_mutex_lock (&ct->lock); + ct->state = state; + ct->msgid = msgid; + ct->aas = aas; + ct->tpg = ct->pp->tpg_id; + ct->timestamp =time (NULL ); +/* Update path-level cache for prioritizer use */ + ct->pp->alua_state = aas; + ct->pp->alua_timestamp = ct->timestamp; +pthread_cond_signal (&ct->active); +pthread_mutex_unlock (&ct->lock); + + condlog(4, "%d:%d : alua checker finished, state %s",
major(ct->devt),libmultipath/checkers/alua.c new file +36
-0+ minor(ct->devt), checker_state_name(state)); + + running =uatomic_xchg (&ct->running,0 ); +if (!running) +pause (); + +alua_thread_cleanup_pop (ct); + +return ((void *)0 ); +} + +static void alua_set_async_timeout (struct checker *c) +{ +struct alua_checker_context *ct = c->context; +struct timespec now; + +get_monotonic_time (&now); + ct->time = now.tv_sec + c->timeout; +} + +static int alua_check_async_timeout (struct checker *c) +{ +struct alua_checker_context *ct = c->context; +struct timespec now; + +get_monotonic_time (&now); +return (now.tv_sec > ct->time); +} + +static int check_pending(struct checker *c) +{ + struct alua_checker_context *ct = c->context; + int alua_status = PATH_PENDING; + + pthread_mutex_lock(&ct->lock); + if (ct->state != PATH_PENDING || ct->msgid !=
MSG_ALUA_RUNNING) {libmultipath/checkers/alua.c new file +5
-0+ alua_status = ct->state; + c->msgid = ct->msgid; + } +pthread_mutex_unlock (&ct->lock); + if (alua_status == PATH_PENDING && c->msgid ==
MSG_ALUA_RUNNING) {libmultipath/checkers/alua.c new file +17
-0+condlog (4 ,"%d:%d : alua checker still running" , +major (ct->devt),minor (ct->devt)); + }else { +int running =uatomic_xchg (&ct->running,0 ); +if (running) +pthread_cancel (ct->thread); + ct->thread =0 ; + } + + ct->checked_state = true; +return alua_status; +} + +bool libcheck_need_wait (struct checker *c) +{ +struct alua_checker_context *ct = c->context; + return (ct && ct->thread && uatomic_read(&ct->running) != 0&&
libmultipath/checkers/alua.c new file +8
-0+ !ct->checked_state); +} + +int libcheck_pending (struct checker *c) +{ +struct alua_checker_context *ct = c->context; + +/* The if path checker isn't running, just return the exiting value. */
libmultipath/checkers/alua.c new file +36
-0+if (!ct || !ct->thread) +return c->path_state; + +return check_pending (c); +} + +int libcheck_check (struct checker *c) +{ +struct alua_checker_context *ct = c->context; + struct path *pp = container_of(c, struct path, checker); +pthread_attr_t attr; +int alua_status =PATH_PENDING , r; +int aas; + +if (!ct) +return PATH_UNCHECKED ; + +if (checker_is_sync (c)) { + alua_status =alua_check (pp, &aas, &c->msgid); + ct->tpg = pp->tpg_id; + ct->aas = aas; + ct->timestamp =time (NULL ); +/* Update path-level cache for prioritizer use */ + pp->alua_state = aas; + pp->alua_timestamp = ct->timestamp; +return alua_status; + } + +/* Async mode */ +if (ct->thread) { + ct->checked_state = true; +if (alua_check_async_timeout (c)) { +int running =uatomic_xchg (&ct->running,0 ); +if (running) { +pthread_cancel (ct->thread); + condlog(3, "%d:%d : alua checkertimeout",
+major(ct->devt), minor(ct-Quoted 1 line
devt));
libmultipath/checkers/alua.c new file +12
-0+ c->msgid =MSG_ALUA_TIMEOUT ; + alua_status =PATH_TIMEOUT ; + }else { +pthread_mutex_lock (&ct->lock); + alua_status = ct->state; + c->msgid = ct->msgid; +pthread_mutex_unlock (&ct->lock); + } + ct->thread =0 ; + }else if (uatomic_read (&ct->running) !=0 ) { +/* +* ct->running may be stale - the thread sets
+ * msgid/state under lock, then clears
running
+ * after releasing it. Check msgid under
lock
+ * to get the truth. OnlyMSG_ALUA_RUNNING
meanslibmultipath/checkers/alua.c new file +20
-0+* no result yet. +*/ +pthread_mutex_lock (&ct->lock); +if (ct->msgid !=MSG_ALUA_RUNNING ) { + alua_status = ct->state; + c->msgid = ct->msgid; + ct->thread =0 ; + } +pthread_mutex_unlock (&ct->lock); + }else { +/* ALUA checker done */ + ct->thread =0 ; +pthread_mutex_lock (&ct->lock); + alua_status = ct->state; + c->msgid = ct->msgid; +pthread_mutex_unlock (&ct->lock); + } + }else { +if (uatomic_read (&ct->holders) >1 ) { +/* The thread has been cancelled but hasn't quit. */
libmultipath/checkers/alua.c new file +2
-0+if (ct->nr_timeouts <=MAX_NR_TIMEOUTS ) { + if (ct->nr_timeouts ==
MAX_NR_TIMEOUTS) {
+ condlog(2, "%d:%d : waiting
for stalled alua thread to finish",
+major(ct->devt),minor(ct->devt));libmultipath/checkers/alua.c new file +8
-0+ } + ct->nr_timeouts++; + c->msgid =MSG_ALUA_TIMEOUT ; +return PATH_TIMEOUT ; + } + ct->nr_timeouts++; +/* +* Start a new thread while the old one is stalled.
+ * We have to prevent it from interfering
with the new
+ * thread. We create a new context and leave
the old
+ * one with the stale thread, hoping it will
clean uplibmultipath/checkers/alua.c new file +3
-0+* eventually. +*/ + condlog(3, "%d:%d : alua thread notresponding",
libmultipath/checkers/alua.c new file +7
-0+ major(ct->devt), minor(ct->devt)); + +if (libcheck_init (c) !=0 ) { + c->msgid =CHECKER_MSGID_DOWN ; +return PATH_UNCHECKED ; + } + ((struct alua_checker_context *)c->context)-Quoted 1 line
nr_timeouts=ct->nr_timeouts;libmultipath/checkers/alua.c new file +5
-0+ +if (!uatomic_sub_return (&ct->holders,1 )) { +/* It did terminate, eventually */ +cleanup_context (ct); + ((struct alua_checker_context *)c-Quoted 1 line
context)->nr_timeouts = 0;
libmultipath/checkers/alua.c new file +20
-0+ } + + ct = c->context; + }else + ct->nr_timeouts =0 ; + +/* Start new ALUA checker */ +pthread_mutex_lock (&ct->lock); + alua_status = ct->state =PATH_PENDING ; + c->msgid = ct->msgid =MSG_ALUA_RUNNING ; +pthread_mutex_unlock (&ct->lock); + ct->fd = c->fd; + ct->timeout = c->timeout; + ct->pp = pp; + ct->checked_state = false; +uatomic_add (&ct->holders,1 ); +uatomic_set (&ct->running,1 ); +alua_set_async_timeout (c); +setup_thread_attr (&attr,32 *1024 ,1 ); + r = start_checker_thread(&ct->thread, &attr, &ct-Quoted 1 line
ctx);
libmultipath/checkers/alua.c new file +6
-0+pthread_attr_destroy (&attr); +if (r) { +uatomic_sub (&ct->holders,1 ); +uatomic_set (&ct->running,0 ); + ct->thread =0 ; + condlog(3, "%d:%d : failed to start aluathread, using"
+ " sync mode",major(ct->devt),minor(ct->devt));
+alua_status= alua_check(pp, &aas, &c-Quoted 1 line
msgid);
libmultipath/checkers/alua.c new file +4
-0+ ct->tpg = pp->tpg_id; + ct->aas = aas; + ct->timestamp =time (NULL ); +/* Update path-level cache for prioritizer use */
libmultipath/checkers/alua.c new file +9
-0+ pp->alua_state = aas; + pp->alua_timestamp = ct->timestamp; +return alua_status; + } + } + +return alua_status; +} +libmultipath/checkers/alua.h new file +0
-0libmultipath/checkers/alua.h line 4
Quoted 4 lines
libmultipath/checkers/alua.h new file +4
-0@@ -0,0 +1,15 @@ +/* +* ALUA Path Checker Header +* +* This file is released under the GPL. Please add a proper license header.
Quoted 13 lines
libmultipath/checkers/alua.h line 15
Quoted 11 lines
libmultipath/checkers/alua.h new file +11
-0+*/ +#ifndef _ALUA_CHECKER_H +#define _ALUA_CHECKER_H + +/* ALUA checker caches state in struct path for prioritizer use: +* pp->alua_state - cached AAS value (-1 if not cached) +* pp->alua_timestamp - when the state was last updated +*/ + +#endif /* _ALUA_CHECKER_H */ +Only comments in this file?
I will look at this. It was likely leftover.
Quoted 35 lines
Quoted 35 lines
libmultipath/prio.c +1
-0@@ -29,6 +29,7 @@ int init_prio(void)#ifdef LOAD_ALL_SHARED_LIBS static const char *const all_prios[] = {PRIO_ALUA , +PRIO_ALUA_CACHED ,PRIO_CONST ,PRIO_DATACORE , PRIO_EMC,libmultipath/prio.h +1
-0@@ -17,6 +17,7 @@ struct path;* Known prioritizers for use in hwtable.c */ #define PRIO_ALUA "alua" +#define PRIO_ALUA_CACHED "alua_cached"#define PRIO_CONST "const"#define PRIO_DATACORE "datacore"#define PRIO_EMC "emc"libmultipath/prioritizers/Makefile +0
-0@@ -13,6 +13,7 @@ LIBDEPS = -lmultipath -lmpathutil -lm -lpthread -lrt
# If you add or remove a prioritizer also update
multipath/multipath.conf.5
Quoted 8 lines
Quoted 6 lines
LIBS = \ libprioalua.so \ + libprioalua_cached.so \ libprioconst.so \ libpriodatacore.so \ libprioemc.so \I think the below should go into a separate patch.
Can they be separated? The prioritizer's logic relies on the checker populating values for its decision.
Quoted 47 lines
Quoted 8 lines
libmultipath/prioritizers/alua_cached.c new file +0
-0libmultipath/prioritizers/alua_cached.c line 8
Quoted 8 lines
libmultipath/prioritizers/alua_cached.c new file +8
-0@@ -0,0 +1,224 @@ +/* +* (C) Copyright IBM Corp. 2004, 2005 All Rights Reserved. +* Modified 2024 to support cached ALUA state from checker +* +* ALUA prioritizer that can use cached state from alua checker +* to avoid duplicate RTPG commands. +* +* This file is released under the GPL. Please add a proper license header.
Quoted 28 lines
libmultipath/prioritizers/alua_cached.c new file +28
-0+*/ +#include <stdio.h> +#include <string.h> + +#include <time.h> + +#include "debug.h" +#include "prio.h" +#include "structs.h" +#include "checkers.h" +#include "discovery.h" + +#include "alua.h" + +#define ALUA_PRIO_NOT_SUPPORTED 1 +#define ALUA_PRIO_RTPG_FAILED 2 +#define ALUA_PRIO_GETAAS_FAILED 3 +#define ALUA_PRIO_TPGS_FAILED 4 +#define ALUA_PRIO_NO_INFORMATION 5 +#define ALUA_PRIO_CACHED_STALE 6 + +/* Maximum age of cached ALUA state in seconds */ +#define ALUA_CACHE_MAX_AGE_SECS 5 + +static const char * aas_string[] = { + [AAS_OPTIMIZED ] ="active/optimized" , + [AAS_NON_OPTIMIZED ] ="active/non-optimized" , + [AAS_STANDBY ] ="standby" ,
Quoted 8 lines
libmultipath/prioritizers/alua_cached.c line 42
Quoted 6 lines
libmultipath/prioritizers/alua_cached.c new file +6
-0+ [AAS_UNAVAILABLE] = "unavailable", + [AAS_LBA_DEPENDENT] = "logical block dependent", + [AAS_RESERVED] = "ARRAY BUG: invalid TPGs state!", + [AAS_OFFLINE] = "offline", + [AAS_TRANSITIONING] = "transitioning between states", +};We shouldn't duplicate the definition of these constants.
This was somewhat intentional though not efficient. I will do better with leveraging in the next patch.
Quoted 371 lines
Quoted 371 lines
libmultipath/prioritizers/alua_cached.c new file +22
-0+ +static const char *aas_print_string (int rc) +{ + rc &=0x7f ; + +if (rc &0x70 ) +return aas_string[AAS_RESERVED ]; + rc &=0x0f ; +if (rc >AAS_RESERVED && rc <AAS_OFFLINE ) +return aas_string[AAS_RESERVED ]; +else +return aas_string[rc]; +} + +/* +* Try to get ALUA info with optimization: +* 1. If using alua checker: use its cached state (it just polled!) +* 2. Try sysfs (kernel maintains this, no I/O needed) +* 3. Fall back to issuing RTPG directly +* +* Rationale: The checker is our active poller. If we're using alua +* and its cache is fresh, we trust that data first. If the cache misses
+ * (stale or empty), we try sysfs before issuing RTPG. This provides a
+ * three-tier optimization: checker cache → sysfs → RTPG.
+ */libmultipath/prioritizers/alua_cached.c new file +14
-0+/* Maximum length of sysfs access_state string */ +#define ALUA_ACCESS_STATE_SIZE 32 + +int +get_alua_info_cached (struct path * pp,bool *used_cache) +{ +int rc; +int tpg; +bool diff_tpg; +char buff[ALUA_ACCESS_STATE_SIZE ]; + + *used_cache = false; + +/* First: if using alua checker, use its fresh data from path structure */
libmultipath/prioritizers/alua_cached.c new file +9
-0+ if (checker_selected(&pp->checker)) { + const char *chk_name = checker_name(&pp->checker); + + if (chk_name && strcmp(chk_name, "alua") == 0 && + pp->alua_state >= 0) { + time_t now = time(NULL); + time_t age = now - pp->alua_timestamp; + + if (pp->alua_timestamp > 0 && age <=
ALUA_CACHE_MAX_AGE_SECS) {
+ condlog(4, "%s: using cached ALUA
state from checker (fresh poll)",pp->dev);libmultipath/prioritizers/alua_cached.c new file +4
-0+ *used_cache = true; +return pp->alua_state; + } + condlog(4, "%s: cached ALUA state notavailable or stale, trying sysfs",
libmultipath/prioritizers/alua_cached.c new file +2
-0+ pp->dev); +/* Cache miss - try sysfs before issuing RTPG */
libmultipath/prioritizers/alua_cached.c new file +4
-0+ } + } + +/* Second: try sysfs (if cache missed or not using alua checker) */
+ rc = sysfs_get_asymmetric_access_state(pp, buff,
sizeof(buff));libmultipath/prioritizers/alua_cached.c new file +16
-0+if (rc >=0 ) { +/* Parse sysfs state string to ALUA state value */ +int aas =-1 ; +int priopath = rc;/* rc contains preferred bit */ + +if (!strncmp (buff,"active/optimized" ,16 )) + aas =AAS_OPTIMIZED ; +else if (!strncmp (buff,"active/non-optimized" ,20 )) + aas =AAS_NON_OPTIMIZED ; +else if (!strncmp (buff,"lba-dependent" ,13 )) + aas =AAS_LBA_DEPENDENT ; +else if (!strncmp (buff,"standby" ,7 )) + aas =AAS_STANDBY ; + +if (aas >=0 ) { + condlog(4, "%s: using sysfs ALUA state (noI/O)",
pp->dev);libmultipath/prioritizers/alua_cached.c new file +16
-0+ *used_cache = true; +return (priopath ?0x80 :0 ) | aas; + } + } + +/* Last resort: issue RTPG directly */ + tpg =get_target_port_group (pp); +if (tpg <0 ) { + rc =get_target_port_group_support (pp); +if (rc <0 ) +return -ALUA_PRIO_TPGS_FAILED ; +if (rc ==TPGS_NONE ) +return -ALUA_PRIO_NOT_SUPPORTED ; +return -ALUA_PRIO_RTPG_FAILED ; + } + diff_tpg = (pp->tpg_id != GROUP_ID_UNDEF && pp->tpg_id !=tpg);
libmultipath/prioritizers/alua_cached.c new file +2
-0+ pp->tpg_id = tpg; + condlog((diff_tpg) ? 2 : 4, "%s: reported target port groupis %i",
libmultipath/prioritizers/alua_cached.c new file +4
-0+ pp->dev, tpg); + rc =get_asymmetric_access_state (pp, tpg); +if (rc <0 ) { + condlog(2, "%s: get_asymmetric_access_state returned%d",
libmultipath/prioritizers/alua_cached.c new file +5
-0+ __func__, rc); +return -ALUA_PRIO_GETAAS_FAILED ; + } + + condlog(3, "%s: aas = %02x [%s]%s", pp->dev, rc,
aas_print_string(rc),libmultipath/prioritizers/alua_cached.c new file +36
-0+ (rc & 0x80) ? " [preferred]" : ""); +return rc; +} + +int get_exclusive_pref_arg (char *args) +{ +char *ptr; + +if (args ==NULL ) +return 0 ; + ptr =strstr (args,"exclusive_pref_bit" ); +if (!ptr) +return 0 ; +if (ptr[18 ] !='\0' && ptr[18 ] !=' ' && ptr[18 ] !='\t' ) +return 0 ; +if (ptr != args && ptr[-1 ] !=' ' && ptr[-1 ] !='\t' ) +return 0 ; +return 1 ; +} + +int getprio (struct path * pp,char * args) +{ +int rc; +int aas; +int priopath; +int exclusive_pref; +bool used_cache = false; + +if (pp->fd <0 ) +return -ALUA_PRIO_NO_INFORMATION ; + + exclusive_pref =get_exclusive_pref_arg (args); + rc =get_alua_info_cached (pp, &used_cache); + +if (used_cache) { + condlog(4, "%s: priority calculated from cached ALUAstate (no RTPG issued)",
libmultipath/prioritizers/alua_cached.c new file +23
-0+ pp->dev); + } + +if (rc >=0 ) { + aas = (rc &0x0f ); + priopath = (rc &0x80 ); +switch (aas) { +case AAS_OPTIMIZED : + rc =50 ; +break ; +case AAS_NON_OPTIMIZED : + rc =10 ; +break ; +case AAS_LBA_DEPENDENT : + rc =5 ; +break ; +case AAS_STANDBY : + rc =1 ; +break ; +default : + rc =0 ; + } + if (priopath && (aas != AAS_OPTIMIZED ||
exclusive_pref))libmultipath/prioritizers/alua_cached.c new file +5
-0+ rc +=80 ; + }else { +switch (-rc) { +case ALUA_PRIO_NOT_SUPPORTED : + condlog(0, "%s: alua not supported", pp-Quoted 1 line
dev);
libmultipath/prioritizers/alua_cached.c new file +3
-0+break ; +case ALUA_PRIO_RTPG_FAILED : + condlog(0, "%s: couldn't get target portgroup",
pp->dev);libmultipath/prioritizers/alua_cached.c new file +3
-0+break ; +case ALUA_PRIO_GETAAS_FAILED : + condlog(0, "%s: couldn't get asymmetricaccess state",
pp->dev);libmultipath/prioritizers/alua_cached.c new file +3
-0+break ; +case ALUA_PRIO_TPGS_FAILED : + condlog(3, "%s: couldn't get supported aluastates",
pp->dev);libmultipath/prioritizers/alua_cached.c new file +6
-0+break ; + } + } +return rc; +} +libmultipath/prioritizers/alua_rtpg.c +7
-0@@ -75,6 +75,7 @@ enum scsi_disposition {SCSI_GOOD =0 ,SCSI_ERROR ,SCSI_RETRY , + SCSI_DISCONNECTED, }; static int@@ -124,6 +125,12 @@ scsi_error(struct sg_io_hdr *hdr, int opcode) PRINT_DEBUG("alua: SCSI error for command %02x: status %02x, sense %02x/%02x/%02x", opcode, hdr->status, sense_key, asc, ascq); +/* Check for disconnected LUN (unmapped at target) */ +if (sense_key ==0x5 && asc ==0x25 && ascq ==0x00 ) { +/* ILLEGAL REQUEST / LOGICAL UNIT NOT SUPPORTED */ +return SCSI_DISCONNECTED ; + } +if (sense_key ==UNIT_ATTENTION || sense_key ==NOT_READY )return SCSI_RETRY ;else
libmultipath/prioritizers/alua_rtpg.c +4
-1@@ -325,7 +332,10 @@ retry: } rc =scsi_error (&hdr,OPERATION_CODE_RTPG ); -if (rc ==SCSI_ERROR ) { +if (rc ==SCSI_DISCONNECTED ) { + PRINT_DEBUG("do_rtpg: LUN disconnected (unmapped at target)!"); + return -RTPG_LUN_DISCONNECTED ; + }else if (rc ==SCSI_ERROR ) {PRINT_DEBUG ("do_rtpg: SCSI error!" );return -RTPG_RTPG_FAILED ;} else if (rc == SCSI_RETRY) {libmultipath/prioritizers/alua_rtpg.h +1
-0@@ -21,6 +21,7 @@#define RTPG_NO_TPG_IDENTIFIER 2#define RTPG_RTPG_FAILED 3#define RTPG_TPG_NOT_FOUND 4 +#define RTPG_LUN_DISCONNECTED 5int get_target_port_group_support (const struct path *pp);int get_target_port_group (const struct path *pp);libmultipath/prioritizers/sysfs.c +5
-2@@ -10,6 +10,9 @@#include "discovery.h" #include "prio.h" +/* Maximum length of sysfs access_state string */ +#define ALUA_ACCESS_STATE_SIZE 32 + static const struct { unsigned char value; char *name;@@ -39,11 +42,11 @@ int get_exclusive_pref_arg(char *args)int getprio (struct path * pp,char *args) {int prio =0 , rc, i; -char buff[512 ]; +char buff[ALUA_ACCESS_STATE_SIZE ];int exclusive_pref; exclusive_pref =get_exclusive_pref_arg (args); - rc =sysfs_get_asymmetric_access_state (pp, buff,512 ); + rc =sysfs_get_asymmetric_access_state (pp, buff,sizeof (buff));if (rc <0 )return PRIO_UNDEF ;prio = 0;libmultipath/propsel.c +1
-0@@ -262,6 +262,7 @@ verify_alua_prio(struct multipath *mp)if (!prio_selected (&pp->prio))continue ;if (strncmp (name,PRIO_ALUA ,PRIO_NAME_LEN ) && + strncmp(name, PRIO_ALUA_CACHED, PRIO_NAME_LEN)&&
strncmp(name, PRIO_SYSFS, PRIO_NAME_LEN)) return false; assume_alua = true;libmultipath/propsel.c +2
-1@@ -690,7 +691,12 @@ int select_checker(struct config *conf, struct path *pp) } (void )path_get_tpgs (pp);if (pp->tpgs !=TPGS_NONE && pp->tpgs !=TPGS_UNDEF ) { - ckr_name =TUR ; +/* +* For ALUA devices (any TPGS support), use alua
+ * checker which combines path checking with
ALUA statelibmultipath/propsel.c +5
-1+* caching for optimal performance. +*/ + ckr_name =ALUA ; goto out; } }@@ -767,8 +773,15 @@ void detect_prio(struct path *pp)if ((tpgs ==TPGS_EXPLICIT || !check_rdac (pp)) &&sysfs_get_asymmetric_access_state (pp, buff,512 ) >=0 ) default_prio =PRIO_SYSFS ; -else +else { default_prio =PRIO_ALUA ; +/* If using alua checker, use cached prioritizer */
libmultipath/propsel.c +2
-0+if (checker_selected (&pp->checker)) { + const char *chk_name =
checker_name(&pp->checker);
+ if (chk_name&& strcmp(chk_name,
ALUA) == 0)
+default_prio=PRIO_ALUA_CACHED;libmultipath/propsel.c +4
-1+ } + }break ;default :return ;@@ -831,7 +844,8 @@ out:/* * fetch tpgs mode for alua, if its not already obtained */ -if (!strncmp (prio_name (p),PRIO_ALUA ,PRIO_NAME_LEN )) { +if (!strncmp (prio_name (p),PRIO_ALUA ,PRIO_NAME_LEN ) || + !strncmp (prio_name (p),PRIO_ALUA_CACHED ,PRIO_NAME_LEN )) {int tpgs =path_get_tpgs (pp);if (tpgs == TPGS_NONE) {libmultipath/structs.c +2
-0@@ -126,6 +126,8 @@ alloc_path (void) pp->fd =-1 ; pp->tpgs =TPGS_UNDEF ; pp->tpg_id =GROUP_ID_UNDEF ; + pp->alua_state =-1 ; + pp->alua_timestamp =0 ; pp->priority =PRIO_UNDEF ; pp->checkint =CHECKINT_UNDEF ;checker_clear (&pp->checker);libmultipath/structs.h +0
-0@@ -444,6 +444,9 @@ struct path {vector hwe;struct gen_path generic_path;int tpg_id;
Quoted 8 lines
Quoted 8 lines
libmultipath/structs.h +2
-0+/* Cached ALUA state from checker for prioritizer use */ +int alua_state;/* AAS value, -1 if not cached */
+time_talua_timestamp; /* Whenalua_statewas last updated
*/enum ioctl_info_states ioctl_info;};
I will put together a 2nd version which addresses these comments. Thanks,
Brian
Signature · 3 lines
Brian Bunker PURE Storage, Inc. [email protected]
Quoted 13 lines · Martin Wilck
libmultipath/checkers/alua.c line 4
Quoted 11 lines · Brian Bunker
libmultipath/checkers/alua.c new file +4
-0@@ -0,0 +1,426 @@ +/* +* ALUA Path Checker +* +* Copyright (c) 2024
There is no Copyright holder.
Quoted 3 lines
libmultipath/checkers/alua.c line 5
Quoted 1 line
libmultipath/checkers/alua.c new file +1
-0+* This file is released under the GPL. Please add a proper license header.
If the code comes from libmultipath/prioritizers/alua*
For consistency with:libmultipath/prioritizers/alua.c: * This file is released under the GPL. libmultipath/prioritizers/alua_rtpg.c: * This file is released under the GPL. libmultipath/prioritizers/alua_rtpg.h: * This file is released under the GPL. libmultipath/prioritizers/alua_spc3.h: * This file is released under the GPL.
Otherwise, the license should be explicitly specified.
Benjamin Marzinski
maintainerQuoted 45 lines · Brian Bunker
Add an 'alua' path checker that uses RTPG to check path state, and an '
alua_cached' prioritizer that uses the checker's cached ALUA state.The checker maps ALUA Asymmetric Access States to path states:
-AAS_OPTIMIZED,AAS_NON_OPTIMIZED->PATH_UP
-AAS_STANDBY->PATH_GHOST
-AAS_TRANSITIONING->PATH_PENDING
-AAS_UNAVAILABLE,AAS_OFFLINE->PATH_DOWNThe
alua_cachedprioritizer first checks for cached state from the alua checker, then falls back to sysfs, then to issuing RTPG directly. This maintains backward compatibility with other checker configurations.When
detect_checkeris enabled, the alua checker is auto-selected for devices with TPGS support. When the alua checker is in use,detect_prioauto-selects thealua_cachedprioritizer.Signed-off-by Brian Bunker <[email protected]>--- libmultipath/Makefile | 5 + libmultipath/checkers.c | 1 + libmultipath/checkers.h | 1 + libmultipath/checkers/Makefile | 3 +- libmultipath/checkers/alua.c | 426 ++++++++++++++++++++++++ libmultipath/checkers/alua.h | 15 + libmultipath/prio.c | 1 + libmultipath/prio.h | 1 + libmultipath/prioritizers/Makefile | 1 + libmultipath/prioritizers/alua_cached.c | 224 +++++++++++++ libmultipath/prioritizers/alua_rtpg.c | 12 +- libmultipath/prioritizers/alua_rtpg.h | 1 + libmultipath/prioritizers/sysfs.c | 7 +- libmultipath/propsel.c | 20 +- libmultipath/structs.c | 2 + libmultipath/structs.h | 3 + 16 files changed, 716 insertions(+), 7 deletions(-) create mode 100644 libmultipath/checkers/alua.c create mode 100644 libmultipath/checkers/alua.h create mode 100644 libmultipath/prioritizers/alua_cached.clibmultipath/Makefile +0
-0
libmultipath/Makefile nvme-lib.o: nvme-lib.c nvme-ioctl.c nvme-ioctl.h · line 45
Quoted 8 lines
libmultipath/Makefile +5
-0@@ -38,6 +38,11 @@ nvme-lib.o: nvme-lib.c nvme-ioctl.c nvme-ioctl.h dict.o: dict.c $(Q)$(CC) $(CPPFLAGS) $(CFLAGS) -Wno-unused-parameter -c -o $@ $< +# checkers/alua.o needs to find headers in parent directory +checkers/alua.o: checkers/alua.c + @echo building $@ because of $? + $(Q)$(CC) $(CPPFLAGS) $(CFLAGS) -I. -c -o $@ $< +
At the risk of sounding stupid, why is this necessary? libmultipath/checkers/Makefile includes libmultipath in CPPFLAGS.
Quoted 247 lines
libmultipath/Makefile +0
-0make_static = $(shell sed '/^static/!s/^\([a-z]\{1,\} \)/static \1/' <$1 >$2) nvme-ioctl.c: nvme/nvme-ioctl.clibmultipath/checkers.c +1
-0@@ -461,6 +461,7 @@ int init_checkers(void)EMC_CLARIION ,READSECTOR0 ,CCISS_TUR , + ALUA_RTPG, };unsigned int i;libmultipath/checkers.h +1
-0@@ -98,6 +98,7 @@ enum path_check_state {#define EMC_CLARIION "emc_clariion"#define READSECTOR0 "readsector0"#define CCISS_TUR "cciss_tur" +#define ALUA "alua"#define NONE "none"#define INVALID "invalid"libmultipath/checkers/Makefile +2
-1@@ -17,7 +17,8 @@ LIBS= \ libcheckdirectio.so \ libcheckemc_clariion.so \ libcheckhp_sw.so \ - libcheckrdac.so + libcheckrdac.so \ + libcheckalua.so all: $(LIBS)libmultipath/checkers/alua.c new file +200
-0@@ -0,0 +1,426 @@ +/* +* ALUA Path Checker +* +* Copyright (c) 2024 +* This file is released under the GPL. +*/ +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <sys/sysmacros.h> +#include <unistd.h> +#include <fcntl.h> +#include <errno.h> +#include <pthread.h> +#include <urcu.h> +#include <urcu/uatomic.h> + +#include "checkers.h" +#include "debug.h" +#include "structs.h" +#include "prio.h" +#include "util.h" +#include "time-util.h" +#include "../prioritizers/alua.h" +#include "../prioritizers/alua_rtpg.h" +#include "alua.h" + +#define MAX_NR_TIMEOUTS 1 + +enum { +MSG_ALUA_RTPG_NOT_SUPPORTED =CHECKER_FIRST_MSGID , +MSG_ALUA_RTPG_TRANSITIONING , +MSG_ALUA_RUNNING , +MSG_ALUA_TIMEOUT , +}; + +#define IDX_ (x) (MSG_ ## x - CHECKER_FIRST_MSGID) +const char *libcheck_msgtable[] = { + [IDX_ (ALUA_RTPG_NOT_SUPPORTED )] =" ALUA not supported" , + [IDX_ (ALUA_RTPG_TRANSITIONING )] =" path is transitioning" , + [IDX_ (ALUA_RUNNING )] =" still running" , + [IDX_ (ALUA_TIMEOUT )] =" timed out" , +NULL , +}; + +struct alua_checker_context { +dev_t devt; +int state; +int running;/* uatomic access only */ +int fd; +unsigned int timeout; +time_t time; +pthread_t thread; +pthread_mutex_t lock; +pthread_cond_t active; +int holders;/* uatomic access only */ +int msgid; +struct checker_context ctx; +unsigned int nr_timeouts; +bool checked_state; +/* ALUA-specific cached data for prioritizer */ +int aas;/* Asymmetric Access State */ +int tpg;/* Target Port Group */ +time_t timestamp;/* When this data was collected */ +/* Data needed for RTPG in thread */ +struct path *pp;/* Path pointer - only valid during check */ +}; + +int libcheck_init (struct checker *c) +{ +struct alua_checker_context *ct; +struct stat sb; + + ct =malloc (sizeof (struct alua_checker_context )); +if (!ct) +return 1 ; +memset (ct,0 ,sizeof (struct alua_checker_context )); + + ct->state =PATH_UNCHECKED ; + ct->fd =-1 ; + ct->aas =-1 ; + ct->tpg =-1 ; +uatomic_set (&ct->holders,1 ); +pthread_cond_init_mono (&ct->active); +pthread_mutex_init (&ct->lock,NULL ); +if (fstat (c->fd, &sb) ==0 ) + ct->devt = sb.st_rdev; + ct->ctx.cls = c->cls; + c->context = ct; + +return 0 ; +} + +static void cleanup_context (struct alua_checker_context *ct) +{ +pthread_mutex_destroy (&ct->lock); +pthread_cond_destroy (&ct->active); +free (ct); +} + +void libcheck_free (struct checker *c) +{ +if (c->context) { +struct alua_checker_context *ct = c->context; +int holders; +int running; + + running =uatomic_xchg (&ct->running,0 ); +if (running) +pthread_cancel (ct->thread); + ct->thread =0 ; + holders =uatomic_sub_return (&ct->holders,1 ); +if (!holders) +cleanup_context (ct); + c->context =NULL ; + } +return ; +} + +/* +* Map ALUA Asymmetric Access State to path state +*/ +static int alua_state_to_path_state (int aas,short *msgid) +{ +switch (aas &0x0f ) { +case AAS_OPTIMIZED : +case AAS_NON_OPTIMIZED : + *msgid =CHECKER_MSGID_UP ; +return PATH_UP ; +case AAS_STANDBY : + *msgid =CHECKER_MSGID_GHOST ; +return PATH_GHOST ; +case AAS_TRANSITIONING : + *msgid =MSG_ALUA_RTPG_TRANSITIONING ; +return PATH_PENDING ; +case AAS_UNAVAILABLE : +case AAS_OFFLINE : +default : + *msgid =CHECKER_MSGID_DOWN ; +return PATH_DOWN ; + } +} + +/* +* Main ALUA check function - uses alua_rtpg.c library. +* This is called either in sync mode or from the async thread. +*/ +static int +alua_check (struct path *pp,int *aas_out,short *msgid) +{ +int aas; + +if (pp->tpg_id ==GROUP_ID_UNDEF ) { + *msgid =CHECKER_MSGID_DOWN ; +return PATH_DOWN ; + } + + aas =get_asymmetric_access_state (pp, pp->tpg_id); +if (aas <0 ) { +if (aas == -RTPG_LUN_DISCONNECTED ) { + *msgid =CHECKER_MSGID_DISCONNECTED ; +return PATH_DISCONNECTED ; + } + *msgid =CHECKER_MSGID_DOWN ; +return PATH_DOWN ; + } + + *aas_out = aas; +return alua_state_to_path_state (aas, msgid); +} + +#define alua_thread_cleanup_push (ct) pthread_cleanup_push(cleanup_func, ct) +#define alua_thread_cleanup_pop (ct) pthread_cleanup_pop(1) + +static void cleanup_func (void *data) +{ +int holders; +struct alua_checker_context *ct = data; + + holders =uatomic_sub_return (&ct->holders,1 ); +if (!holders) +cleanup_context (ct); +} + +void *libcheck_thread (struct checker_context *ctx) +{ +struct alua_checker_context *ct = + container_of(ctx, struct alua_checker_context, ctx); +int state, running; +short msgid; +int aas; + +/* This thread can be canceled, so setup clean up */ +alua_thread_cleanup_push (ct); + +condlog (4 ,"%d:%d : alua checker starting up" ,major (ct->devt), +minor (ct->devt)); +
libmultipath/checkers/alua.c line 210
Quoted 10 lines
libmultipath/checkers/alua.c new file +10
-0+ state =alua_check (ct->pp, &aas, &msgid); +pthread_testcancel (); + +/* ALUA checker done */ +pthread_mutex_lock (&ct->lock); + ct->state = state; + ct->msgid = msgid; + ct->aas = aas; + ct->tpg = ct->pp->tpg_id; + ct->timestamp =time (NULL );
The multipath code pretty much exclusively uses get_monotonic_time() to track the time. That can get used here as well.
libmultipath/checkers/alua.c line 213
Quoted 3 lines
libmultipath/checkers/alua.c new file +3
-0+/* Update path-level cache for prioritizer use */ + ct->pp->alua_state = aas; + ct->pp->alua_timestamp = ct->timestamp;
This is not safe. The checker_context was specifically designed so that it was possible to free a path while an async checker was still running. The path holds a reference to the context (ct->holders), as does the thread. When the last reference is dropped, the context is freed. So there's no way you can know that the path still exists while the thread is running. The only place where it is safe to access the path is in libcheck_check().
Quoted 82 lines
libmultipath/checkers/alua.c new file +82
-0+pthread_cond_signal (&ct->active); +pthread_mutex_unlock (&ct->lock); + +condlog (4 ,"%d:%d : alua checker finished, state %s" ,major (ct->devt), +minor (ct->devt),checker_state_name (state)); + + running =uatomic_xchg (&ct->running,0 ); +if (!running) +pause (); + +alua_thread_cleanup_pop (ct); + +return ((void *)0 ); +} + +static void alua_set_async_timeout (struct checker *c) +{ +struct alua_checker_context *ct = c->context; +struct timespec now; + +get_monotonic_time (&now); + ct->time = now.tv_sec + c->timeout; +} + +static int alua_check_async_timeout (struct checker *c) +{ +struct alua_checker_context *ct = c->context; +struct timespec now; + +get_monotonic_time (&now); +return (now.tv_sec > ct->time); +} + +static int check_pending (struct checker *c) +{ +struct alua_checker_context *ct = c->context; +int alua_status =PATH_PENDING ; + +pthread_mutex_lock (&ct->lock); +if (ct->state !=PATH_PENDING || ct->msgid !=MSG_ALUA_RUNNING ) { + alua_status = ct->state; + c->msgid = ct->msgid; + } +pthread_mutex_unlock (&ct->lock); +if (alua_status ==PATH_PENDING && c->msgid ==MSG_ALUA_RUNNING ) { +condlog (4 ,"%d:%d : alua checker still running" , +major (ct->devt),minor (ct->devt)); + }else { +int running =uatomic_xchg (&ct->running,0 ); +if (running) +pthread_cancel (ct->thread); + ct->thread =0 ; + } + + ct->checked_state = true; +return alua_status; +} + +bool libcheck_need_wait (struct checker *c) +{ +struct alua_checker_context *ct = c->context; +return (ct && ct->thread &&uatomic_read (&ct->running) !=0 && + !ct->checked_state); +} + +int libcheck_pending (struct checker *c) +{ +struct alua_checker_context *ct = c->context; + +/* The if path checker isn't running, just return the exiting value. */ +if (!ct || !ct->thread) +return c->path_state; + +return check_pending (c); +} + +int libcheck_check (struct checker *c) +{ +struct alua_checker_context *ct = c->context; + struct path *pp = container_of(c, struct path, checker); +pthread_attr_t attr; +int alua_status =PATH_PENDING , r;
libmultipath/checkers/alua.c line 305
Quoted 10 lines
libmultipath/checkers/alua.c new file +10
-0+int aas; + +if (!ct) +return PATH_UNCHECKED ; + +if (checker_is_sync (c)) { + alua_status =alua_check (pp, &aas, &c->msgid); + ct->tpg = pp->tpg_id; + ct->aas = aas; + ct->timestamp =time (NULL );
The purpose of the context is to be able to copy data from the thread back to the path checker. I don't see the need to update it when you aren't in the thread. And again, we can use get_monotonic_time() to get the time here.
Quoted 22 lines
libmultipath/checkers/alua.c new file +22
-0+/* Update path-level cache for prioritizer use */ + pp->alua_state = aas; + pp->alua_timestamp = ct->timestamp; +return alua_status; + } + +/* Async mode */ +if (ct->thread) { + ct->checked_state = true; +if (alua_check_async_timeout (c)) { +int running =uatomic_xchg (&ct->running,0 ); +if (running) { +pthread_cancel (ct->thread); +condlog (3 ,"%d:%d : alua checker timeout" , +major (ct->devt),minor (ct->devt)); + c->msgid =MSG_ALUA_TIMEOUT ; + alua_status =PATH_TIMEOUT ; + }else { +pthread_mutex_lock (&ct->lock); + alua_status = ct->state; + c->msgid = ct->msgid; +pthread_mutex_unlock (&ct->lock);
libmultipath/checkers/alua.c line 337
Quoted 10 lines
libmultipath/checkers/alua.c new file +10
-0+ } + ct->thread =0 ; + }else if (uatomic_read (&ct->running) !=0 ) { +/* +* ct->running may be stale - the thread sets +* msgid/state under lock, then clears running +* after releasing it. Check msgid under lock +* to get the truth. Only MSG_ALUA_RUNNING means +* no result yet. +*/
How hard is it to trigger the checker hang you saw with this checker. I'd really like to understand why it happens because I still can't see the necessity of this.
Quoted 9 lines
libmultipath/checkers/alua.c new file +9
-0+pthread_mutex_lock (&ct->lock); +if (ct->msgid !=MSG_ALUA_RUNNING ) { + alua_status = ct->state; + c->msgid = ct->msgid; + ct->thread =0 ; + } +pthread_mutex_unlock (&ct->lock); + }else { +/* ALUA checker done */
libmultipath/checkers/alua.c line 356
Quoted 10 lines
libmultipath/checkers/alua.c new file +10
-0+ ct->thread =0 ; +pthread_mutex_lock (&ct->lock); + alua_status = ct->state; + c->msgid = ct->msgid; +pthread_mutex_unlock (&ct->lock); + } + }else { +if (uatomic_read (&ct->holders) >1 ) { +/* The thread has been cancelled but hasn't quit. */ +if (ct->nr_timeouts <=MAX_NR_TIMEOUTS ) {
I think you meant ">=" here, instead of "<=".
Quoted 50 lines
libmultipath/checkers/alua.c new file +50
-0+if (ct->nr_timeouts ==MAX_NR_TIMEOUTS ) { +condlog (2 ,"%d:%d : waiting for stalled alua thread to finish" , +major (ct->devt),minor (ct->devt)); + } + ct->nr_timeouts++; + c->msgid =MSG_ALUA_TIMEOUT ; +return PATH_TIMEOUT ; + } + ct->nr_timeouts++; +/* +* Start a new thread while the old one is stalled. +* We have to prevent it from interfering with the new +* thread. We create a new context and leave the old +* one with the stale thread, hoping it will clean up +* eventually. +*/ +condlog (3 ,"%d:%d : alua thread not responding" , +major (ct->devt),minor (ct->devt)); + +if (libcheck_init (c) !=0 ) { + c->msgid =CHECKER_MSGID_DOWN ; +return PATH_UNCHECKED ; + } + ((struct alua_checker_context *)c->context)->nr_timeouts = ct->nr_timeouts; + +if (!uatomic_sub_return (&ct->holders,1 )) { +/* It did terminate, eventually */ +cleanup_context (ct); + ((struct alua_checker_context *)c->context)->nr_timeouts =0 ; + } + + ct = c->context; + }else + ct->nr_timeouts =0 ; + +/* Start new ALUA checker */ +pthread_mutex_lock (&ct->lock); + alua_status = ct->state =PATH_PENDING ; + c->msgid = ct->msgid =MSG_ALUA_RUNNING ; +pthread_mutex_unlock (&ct->lock); + ct->fd = c->fd; + ct->timeout = c->timeout; + ct->pp = pp; + ct->checked_state = false; +uatomic_add (&ct->holders,1 ); +uatomic_set (&ct->running,1 ); +alua_set_async_timeout (c); +setup_thread_attr (&attr,32 *1024 ,1 ); + r =start_checker_thread (&ct->thread, &attr, &ct->ctx); +pthread_attr_destroy (&attr);
libmultipath/checkers/alua.c line 416
Quoted 10 lines
libmultipath/checkers/alua.c new file +10
-0+if (r) { +uatomic_sub (&ct->holders,1 ); +uatomic_set (&ct->running,0 ); + ct->thread =0 ; +condlog (3 ,"%d:%d : failed to start alua thread, using" +" sync mode" ,major (ct->devt),minor (ct->devt)); + alua_status =alua_check (pp, &aas, &c->msgid); + ct->tpg = pp->tpg_id; + ct->aas = aas; + ct->timestamp =time (NULL );
Again with not needing to update ct here and using get_monotonic_time.
libmultipath/prioritizers/alua_cached.c
Quoted 72 lines
libmultipath/checkers/alua.c new file +10
-0+/* Update path-level cache for prioritizer use */ + pp->alua_state = aas; + pp->alua_timestamp = ct->timestamp; +return alua_status; + } + } + +return alua_status; +} +libmultipath/checkers/alua.h new file +15
-0@@ -0,0 +1,15 @@ +/* +* ALUA Path Checker Header +* +* This file is released under the GPL. +*/ +#ifndef _ALUA_CHECKER_H +#define _ALUA_CHECKER_H + +/* ALUA checker caches state in struct path for prioritizer use: +* pp->alua_state - cached AAS value (-1 if not cached) +* pp->alua_timestamp - when the state was last updated +*/ + +#endif /* _ALUA_CHECKER_H */ +libmultipath/prio.c +1
-0@@ -29,6 +29,7 @@ int init_prio(void)#ifdef LOAD_ALL_SHARED_LIBS static const char *const all_prios[] = {PRIO_ALUA , +PRIO_ALUA_CACHED ,PRIO_CONST ,PRIO_DATACORE , PRIO_EMC,libmultipath/prio.h +1
-0@@ -17,6 +17,7 @@ struct path;* Known prioritizers for use in hwtable.c */ #define PRIO_ALUA "alua" +#define PRIO_ALUA_CACHED "alua_cached"#define PRIO_CONST "const"#define PRIO_DATACORE "datacore"#define PRIO_EMC "emc"libmultipath/prioritizers/Makefile +1
-0@@ -13,6 +13,7 @@ LIBDEPS = -lmultipath -lmpathutil -lm -lpthread -lrt # If you add or remove a prioritizer also update multipath/multipath.conf.5 LIBS = \ libprioalua.so \ + libprioalua_cached.so \ libprioconst.so \ libpriodatacore.so \ libprioemc.so \libmultipath/prioritizers/alua_cached.c new file +0
-0
There is so much duplication in these ALUA checkers, I am a little perplexed about what to do with this. It makes me wonder if we should just have one alua prioritzer that can use all the sources like this, and the different config options would just control which methods the prioritizer would use.
Quoted 78 lines
libmultipath/prioritizers/alua_cached.c new file +77
-0@@ -0,0 +1,224 @@ +/* +* (C) Copyright IBM Corp. 2004, 2005 All Rights Reserved. +* Modified 2024 to support cached ALUA state from checker +* +* ALUA prioritizer that can use cached state from alua checker +* to avoid duplicate RTPG commands. +* +* This file is released under the GPL. +*/ +#include <stdio.h> +#include <string.h> + +#include <time.h> + +#include "debug.h" +#include "prio.h" +#include "structs.h" +#include "checkers.h" +#include "discovery.h" + +#include "alua.h" + +#define ALUA_PRIO_NOT_SUPPORTED 1 +#define ALUA_PRIO_RTPG_FAILED 2 +#define ALUA_PRIO_GETAAS_FAILED 3 +#define ALUA_PRIO_TPGS_FAILED 4 +#define ALUA_PRIO_NO_INFORMATION 5 +#define ALUA_PRIO_CACHED_STALE 6 + +/* Maximum age of cached ALUA state in seconds */ +#define ALUA_CACHE_MAX_AGE_SECS 5 + +static const char * aas_string[] = { + [AAS_OPTIMIZED ] ="active/optimized" , + [AAS_NON_OPTIMIZED ] ="active/non-optimized" , + [AAS_STANDBY ] ="standby" , + [AAS_UNAVAILABLE ] ="unavailable" , + [AAS_LBA_DEPENDENT ] ="logical block dependent" , + [AAS_RESERVED ] ="ARRAY BUG: invalid TPGs state!" , + [AAS_OFFLINE ] ="offline" , + [AAS_TRANSITIONING ] ="transitioning between states" , +}; + +static const char *aas_print_string (int rc) +{ + rc &=0x7f ; + +if (rc &0x70 ) +return aas_string[AAS_RESERVED ]; + rc &=0x0f ; +if (rc >AAS_RESERVED && rc <AAS_OFFLINE ) +return aas_string[AAS_RESERVED ]; +else +return aas_string[rc]; +} + +/* +* Try to get ALUA info with optimization: +* 1. If using alua checker: use its cached state (it just polled!) +* 2. Try sysfs (kernel maintains this, no I/O needed) +* 3. Fall back to issuing RTPG directly +* +* Rationale: The checker is our active poller. If we're using alua +* and its cache is fresh, we trust that data first. If the cache misses +* (stale or empty), we try sysfs before issuing RTPG. This provides a +* three-tier optimization: checker cache → sysfs → RTPG. +*/ +/* Maximum length of sysfs access_state string */ +#define ALUA_ACCESS_STATE_SIZE 32 + +int +get_alua_info_cached (struct path * pp,bool *used_cache) +{ +int rc; +int tpg; +bool diff_tpg; +char buff[ALUA_ACCESS_STATE_SIZE ];
libmultipath/prioritizers/alua_cached.c line 87
Quoted 10 lines
libmultipath/prioritizers/alua_cached.c new file +10
-0+ + *used_cache = false; + +/* First: if using alua checker, use its fresh data from path structure */ +if (checker_selected (&pp->checker)) { +const char *chk_name =checker_name (&pp->checker); + +if (chk_name &&strcmp (chk_name,"alua" ) ==0 && + pp->alua_state >=0 ) { +time_t now =time (NULL );
Again, you can use get_monotonic_time() & timespecsub() to check the difference.
Quoted 150 lines
libmultipath/prioritizers/alua_cached.c new file +137
-0+time_t age = now - pp->alua_timestamp; + +if (pp->alua_timestamp >0 && age <=ALUA_CACHE_MAX_AGE_SECS ) { +condlog (4 ,"%s: using cached ALUA state from checker (fresh poll)" , pp->dev); + *used_cache = true; +return pp->alua_state; + } +condlog (4 ,"%s: cached ALUA state not available or stale, trying sysfs" , + pp->dev); +/* Cache miss - try sysfs before issuing RTPG */ + } + } + +/* Second: try sysfs (if cache missed or not using alua checker) */ + rc =sysfs_get_asymmetric_access_state (pp, buff,sizeof (buff)); +if (rc >=0 ) { +/* Parse sysfs state string to ALUA state value */ +int aas =-1 ; +int priopath = rc;/* rc contains preferred bit */ + +if (!strncmp (buff,"active/optimized" ,16 )) + aas =AAS_OPTIMIZED ; +else if (!strncmp (buff,"active/non-optimized" ,20 )) + aas =AAS_NON_OPTIMIZED ; +else if (!strncmp (buff,"lba-dependent" ,13 )) + aas =AAS_LBA_DEPENDENT ; +else if (!strncmp (buff,"standby" ,7 )) + aas =AAS_STANDBY ; + +if (aas >=0 ) { +condlog (4 ,"%s: using sysfs ALUA state (no I/O)" , pp->dev); + *used_cache = true; +return (priopath ?0x80 :0 ) | aas; + } + } + +/* Last resort: issue RTPG directly */ + tpg =get_target_port_group (pp); +if (tpg <0 ) { + rc =get_target_port_group_support (pp); +if (rc <0 ) +return -ALUA_PRIO_TPGS_FAILED ; +if (rc ==TPGS_NONE ) +return -ALUA_PRIO_NOT_SUPPORTED ; +return -ALUA_PRIO_RTPG_FAILED ; + } + diff_tpg = (pp->tpg_id !=GROUP_ID_UNDEF && pp->tpg_id != tpg); + pp->tpg_id = tpg; +condlog ((diff_tpg) ?2 :4 ,"%s: reported target port group is %i" , + pp->dev, tpg); + rc =get_asymmetric_access_state (pp, tpg); +if (rc <0 ) { +condlog (2 ,"%s: get_asymmetric_access_state returned %d" , + __func__, rc); +return -ALUA_PRIO_GETAAS_FAILED ; + } + +condlog (3 ,"%s: aas = %02x [%s]%s" , pp->dev, rc,aas_print_string (rc), + (rc &0x80 ) ?" [preferred]" :"" ); +return rc; +} + +int get_exclusive_pref_arg (char *args) +{ +char *ptr; + +if (args ==NULL ) +return 0 ; + ptr =strstr (args,"exclusive_pref_bit" ); +if (!ptr) +return 0 ; +if (ptr[18 ] !='\0' && ptr[18 ] !=' ' && ptr[18 ] !='\t' ) +return 0 ; +if (ptr != args && ptr[-1 ] !=' ' && ptr[-1 ] !='\t' ) +return 0 ; +return 1 ; +} + +int getprio (struct path * pp,char * args) +{ +int rc; +int aas; +int priopath; +int exclusive_pref; +bool used_cache = false; + +if (pp->fd <0 ) +return -ALUA_PRIO_NO_INFORMATION ; + + exclusive_pref =get_exclusive_pref_arg (args); + rc =get_alua_info_cached (pp, &used_cache); + +if (used_cache) { +condlog (4 ,"%s: priority calculated from cached ALUA state (no RTPG issued)" , + pp->dev); + } + +if (rc >=0 ) { + aas = (rc &0x0f ); + priopath = (rc &0x80 ); +switch (aas) { +case AAS_OPTIMIZED : + rc =50 ; +break ; +case AAS_NON_OPTIMIZED : + rc =10 ; +break ; +case AAS_LBA_DEPENDENT : + rc =5 ; +break ; +case AAS_STANDBY : + rc =1 ; +break ; +default : + rc =0 ; + } +if (priopath && (aas !=AAS_OPTIMIZED || exclusive_pref)) + rc +=80 ; + }else { +switch (-rc) { +case ALUA_PRIO_NOT_SUPPORTED : +condlog (0 ,"%s: alua not supported" , pp->dev); +break ; +case ALUA_PRIO_RTPG_FAILED : +condlog (0 ,"%s: couldn't get target port group" , pp->dev); +break ; +case ALUA_PRIO_GETAAS_FAILED : +condlog (0 ,"%s: couldn't get asymmetric access state" , pp->dev); +break ; +case ALUA_PRIO_TPGS_FAILED : +condlog (3 ,"%s: couldn't get supported alua states" , pp->dev); +break ; + } + } +return rc; +} +libmultipath/prioritizers/alua_rtpg.c +1
-0@@ -75,6 +75,7 @@ enum scsi_disposition {SCSI_GOOD =0 ,SCSI_ERROR ,SCSI_RETRY , + SCSI_DISCONNECTED, }; static int
libmultipath/prioritizers/alua_rtpg.c scsi_error(struct sg_io_hdr *hdr, int opcode) · line 133
Quoted 9 lines
libmultipath/prioritizers/alua_rtpg.c +6
-0@@ -124,6 +125,12 @@ scsi_error(struct sg_io_hdr *hdr, int opcode)PRINT_DEBUG ("alua: SCSI error for command %02x: status %02x, sense %02x/%02x/%02x" , opcode, hdr->status, sense_key, asc, ascq); +/* Check for disconnected LUN (unmapped at target) */ +if (sense_key ==0x5 && asc ==0x25 && ascq ==0x00 ) { +/* ILLEGAL REQUEST / LOGICAL UNIT NOT SUPPORTED */ +return SCSI_DISCONNECTED ; + } +
You need to make sure to handle SCSI_DISCONNECTED in do_inquiry_sg(), which also calls scsi_error().
Quoted 71 lines
libmultipath/prioritizers/alua_rtpg.c +4
-1if (sense_key ==UNIT_ATTENTION || sense_key ==NOT_READY )return SCSI_RETRY ;else @@ -325,7 +332,10 @@ retry: } rc =scsi_error (&hdr,OPERATION_CODE_RTPG ); -if (rc ==SCSI_ERROR ) { +if (rc ==SCSI_DISCONNECTED ) { +PRINT_DEBUG ("do_rtpg: LUN disconnected (unmapped at target)!" ); +return -RTPG_LUN_DISCONNECTED ; + }else if (rc ==SCSI_ERROR ) {PRINT_DEBUG ("do_rtpg: SCSI error!" );return -RTPG_RTPG_FAILED ; }else if (rc ==SCSI_RETRY ) {libmultipath/prioritizers/alua_rtpg.h +1
-0@@ -21,6 +21,7 @@#define RTPG_NO_TPG_IDENTIFIER 2#define RTPG_RTPG_FAILED 3#define RTPG_TPG_NOT_FOUND 4 +#define RTPG_LUN_DISCONNECTED 5int get_target_port_group_support (const struct path *pp);int get_target_port_group (const struct path *pp);libmultipath/prioritizers/sysfs.c +5
-2@@ -10,6 +10,9 @@#include "discovery.h" #include "prio.h" +/* Maximum length of sysfs access_state string */ +#define ALUA_ACCESS_STATE_SIZE 32 + static const struct { unsigned char value; char *name;@@ -39,11 +42,11 @@ int get_exclusive_pref_arg(char *args)int getprio (struct path * pp,char *args) {int prio =0 , rc, i; -char buff[512 ]; +char buff[ALUA_ACCESS_STATE_SIZE ];int exclusive_pref; exclusive_pref =get_exclusive_pref_arg (args); - rc =sysfs_get_asymmetric_access_state (pp, buff,512 ); + rc =sysfs_get_asymmetric_access_state (pp, buff,sizeof (buff));if (rc <0 )return PRIO_UNDEF ; prio =0 ;libmultipath/propsel.c +1
-0@@ -262,6 +262,7 @@ verify_alua_prio(struct multipath *mp)if (!prio_selected (&pp->prio))continue ;if (strncmp (name,PRIO_ALUA ,PRIO_NAME_LEN ) && +strncmp (name,PRIO_ALUA_CACHED ,PRIO_NAME_LEN ) &&strncmp (name,PRIO_SYSFS ,PRIO_NAME_LEN ))return false; assume_alua = true;@@ -690,7 +691,12 @@ int select_checker(struct config *conf, struct path *pp) } (void )path_get_tpgs (pp);if (pp->tpgs !=TPGS_NONE && pp->tpgs !=TPGS_UNDEF ) {
libmultipath/propsel.c int select_checker(struct config *conf, struct path *pp) · line 702
Quoted 10 lines
libmultipath/propsel.c +6
-1- ckr_name =TUR ; +/* +* For ALUA devices (any TPGS support), use alua +* checker which combines path checking with ALUA state +* caching for optimal performance. +*/ + ckr_name =ALUA ; goto out; } }
I think we should actually have a separate option to distinguish between ALUA and TUR as the default autodetected checker for ALUA devices. Something like detect_pgpolicy_use_tpg does. That way there is just one config option that needs to get changed to switch between the new and old behavior.
-Ben
Quoted 56 lines
libmultipath/propsel.c +10
-2@@ -767,8 +773,15 @@ void detect_prio(struct path *pp)if ((tpgs ==TPGS_EXPLICIT || !check_rdac (pp)) &&sysfs_get_asymmetric_access_state (pp, buff,512 ) >=0 ) default_prio =PRIO_SYSFS ; -else +else { default_prio =PRIO_ALUA ; +/* If using alua checker, use cached prioritizer */ +if (checker_selected (&pp->checker)) { +const char *chk_name =checker_name (&pp->checker); +if (chk_name &&strcmp (chk_name,ALUA ) ==0 ) + default_prio =PRIO_ALUA_CACHED ; + } + }break ;default :return ;@@ -831,7 +844,8 @@ out:/* * fetch tpgs mode for alua, if its not already obtained */ -if (!strncmp (prio_name (p),PRIO_ALUA ,PRIO_NAME_LEN )) { +if (!strncmp (prio_name (p),PRIO_ALUA ,PRIO_NAME_LEN ) || + !strncmp (prio_name (p),PRIO_ALUA_CACHED ,PRIO_NAME_LEN )) {int tpgs =path_get_tpgs (pp);if (tpgs ==TPGS_NONE ) {libmultipath/structs.c +2
-0@@ -126,6 +126,8 @@ alloc_path (void) pp->fd =-1 ; pp->tpgs =TPGS_UNDEF ; pp->tpg_id =GROUP_ID_UNDEF ; + pp->alua_state =-1 ; + pp->alua_timestamp =0 ; pp->priority =PRIO_UNDEF ; pp->checkint =CHECKINT_UNDEF ;checker_clear (&pp->checker);libmultipath/structs.h +3
-0@@ -444,6 +444,9 @@ struct path {vector hwe;struct gen_path generic_path;int tpg_id; +/* Cached ALUA state from checker for prioritizer use */ +int alua_state;/* AAS value, -1 if not cached */ +time_t alua_timestamp;/* When alua_state was last updated */ enum ioctl_info_states ioctl_info; };Signature · 1 line
2.50.1 (Apple Git-155)
Quoted 54 lines · Brian Bunker
For ALUA-capable storage arrays, multipath-tools currently uses TUR (Test Unit Ready) as the default path checker while a separate prioritizer determines ALUA state. When sysfs provides ALUA state (the common case), no RTPG command is needed for priority. However, if sysfs is unavailable or
detect_priois off, the prioritizer issues RTPG, resulting in two SCSI commands per path check cycle: TUR + RTPG.This patch introduces an 'alua' path checker that uses RTPG as the path check command, providing two benefits:
1. For configurations where RTPG is needed for priority, this eliminates duplicate I/O by combining path checking and ALUA state retrieval into a single RTPG command.
2. RTPG provides richer path state information than TUR:
* TRANSITIONING state detected and mapped toPATH_PENDING, preventing false all paths down during controller failover
* STANDBY state detected and mapped toPATH_GHOST
* UNAVAILABLE/OFFLINE states properly distinguishedThe richer state information is valuable even when sysfs is available, as TUR cannot distinguish between these ALUA-specific conditions.
This design intentionally couples the checker and prioritizer: the
alua_cachedprioritizer consumes state cached by the alua checker. While this blurs the traditional separation between these components, for ALUA the path state and priority fundamentally derive from the same underlying data, making the coupling a natural fit.This patch sets the alua checker as the auto-detected default for ALUA devices to demonstrate its benefits. We recognize that adopting a new default checker requires careful consideration; the auto-detection can be adjusted or removed based on review feedback.
Brian Bunker (1):
checkers: add alua path checkerlibmultipath/Makefile | 5 + libmultipath/checkers.c | 1 + libmultipath/checkers.h | 1 + libmultipath/checkers/Makefile | 3 +- libmultipath/checkers/alua.c | 426 ++++++++++++++++++++++++ libmultipath/checkers/alua.h | 15 + libmultipath/prio.c | 1 + libmultipath/prio.h | 1 + libmultipath/prioritizers/Makefile | 1 + libmultipath/prioritizers/alua_cached.c | 225 +++++++++++++ libmultipath/prioritizers/alua_rtpg.c | 12 +- libmultipath/prioritizers/alua_rtpg.h | 1 + libmultipath/prioritizers/sysfs.c | 7 +- libmultipath/propsel.c | 25 +- libmultipath/structs.c | 2 + libmultipath/structs.h | 3 + 16 files changed, 721 insertions(+), 8 deletions(-)
Quoted 4 lines
create mode 100644
libmultipath/checkers/alua.ccreate mode 100644libmultipath/checkers/alua.hcreate mode 100644libmultipath/prioritizers/alua_cached.c
Hmm. Some array vendor have been less than happy when I tried a similar thing during development of the ALUA device handler.
Problem is that RTPG requires a cluster callout on the array side, as it needs to contain information about the _entire_ target system. Which, for distributed systems, is a costly operation. So using this as a path checker is not making certain array vendors happy.
However: as you might have followed the discussion about scsi multipathing, maybe we can turn this around to profit both sides:
Can't we move ALUA state handling into the generic code, leaving the device handler only to deal with explicit ALUA?
That way we can intercept sense codes and send RTPG only when required, such that the ALUA state of the scsi device is correct.
And the the ALUA path checker just has to evaluate the information in sysfs.
_And_ scsi multipathing can use that information, too.
Hmm?
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Quoted 11 lines · Hannes Reinecke
Quoted 1 line
Hmm. Some array vendor have been less than happy when I tried a similar thing during development of the ALUA device handler.
Problem is that RTPG requires a cluster callout on the array side, as it needs to contain information about the _entire_ target system. Which, for distributed systems, is a costly operation. So using this as a path checker is not making certain array vendors happy.
I wonder if we could avoid this problem by listening to SDEV_EVT_ALUA_STATE_CHANGE_REPORTED uevents in multipathd.
Actually, multipathd could use TUR for checking unless we receive an event of this type. multipathd could listen to those events and then retrieve the new device state(s) from sysfs, without sending an RTPG command itself.
We wouldn't switch to the alua checker by default anyway, so the vendors that prefer the sysfs prioritizer won't be hurt even if that doesn't work.
Quoted 6 lines
However: as you might have followed the discussion about scsi multipathing, maybe we can turn this around to profit both sides:
Can't we move ALUA state handling into the generic code, leaving the device handler only to deal with explicit ALUA?
When you say "generic code", I suppose you are talking about the SCSI midlayer. It isn't clear to me how multipathd would benefit from handling ALUA there, other than being able to read state from sysfs, as it already does.
Regards
Martin
Quoted 12 lines
That way we can intercept sense codes and send RTPG only when required, such that the ALUA state of the scsi device is correct.
And the the ALUA path checker just has to evaluate the information in sysfs.
_And_ scsi multipathing can use that information, too.Hmm?
Cheers,
Hannes
Quoted 8 lines · Martin Wilck
Actually, multipathd could use TUR for checking unless we receive an event of this type. multipathd could listen to those events and then retrieve the new device state(s) from sysfs, without sending an RTPG command itself.
We wouldn't switch to the alua checker by default anyway, so the vendors that prefer the sysfs prioritizer won't be hurt even if that doesn't work.
Just one observation: currently there are ALUA arrays (NetApp E/EF, Dell Unity) where their own checker is preferred.
As well as software-defined storage (Hitachi Vantara VSP One SDS Block, Linux-IO (LIO) Target) where directio must be used.
Quoted 61 lines · Hannes Reinecke
Quoted 54 lines · Brian Bunker
For ALUA-capable storage arrays, multipath-tools currently uses TUR (Test Unit Ready) as the default path checker while a separate prioritizer determines ALUA state. When sysfs provides ALUA state (the common case), no RTPG command is needed for priority. However, if sysfs is unavailable or
detect_priois off, the prioritizer issues RTPG, resulting in two SCSI commands per path check cycle: TUR + RTPG.This patch introduces an 'alua' path checker that uses RTPG as the path check command, providing two benefits:
1. For configurations where RTPG is needed for priority, this eliminates duplicate I/O by combining path checking and ALUA state retrieval into a single RTPG command.
2. RTPG provides richer path state information than TUR:
* TRANSITIONING state detected and mapped toPATH_PENDING, preventing false all paths down during controller failover
* STANDBY state detected and mapped toPATH_GHOST
* UNAVAILABLE/OFFLINE states properly distinguishedThe richer state information is valuable even when sysfs is available, as TUR cannot distinguish between these ALUA-specific conditions.
This design intentionally couples the checker and prioritizer: the
alua_cachedprioritizer consumes state cached by the alua checker. While this blurs the traditional separation between these components, for ALUA the path state and priority fundamentally derive from the same underlying data, making the coupling a natural fit.This patch sets the alua checker as the auto-detected default for ALUA devices to demonstrate its benefits. We recognize that adopting a new default checker requires careful consideration; the auto-detection can be adjusted or removed based on review feedback.
Brian Bunker (1):
checkers: add alua path checkerlibmultipath/Makefile | 5 + libmultipath/checkers.c | 1 + libmultipath/checkers.h | 1 + libmultipath/checkers/Makefile | 3 +- libmultipath/checkers/alua.c | 426 ++++++++++++++++++++++++ libmultipath/checkers/alua.h | 15 + libmultipath/prio.c | 1 + libmultipath/prio.h | 1 + libmultipath/prioritizers/Makefile | 1 + libmultipath/prioritizers/alua_cached.c | 225 +++++++++++++ libmultipath/prioritizers/alua_rtpg.c | 12 +- libmultipath/prioritizers/alua_rtpg.h | 1 + libmultipath/prioritizers/sysfs.c | 7 +- libmultipath/propsel.c | 25 +- libmultipath/structs.c | 2 + libmultipath/structs.h | 3 + 16 files changed, 721 insertions(+), 8 deletions(-)Quoted 4 lines
create mode 100644
libmultipath/checkers/alua.ccreate mode 100644libmultipath/checkers/alua.hcreate mode 100644libmultipath/prioritizers/alua_cached.c
Quoted 8 lines
Hmm. Some array vendor have been less than happy when I tried a similar thing during development of the ALUA device handler.
Problem is that RTPG requires a cluster callout on the array side, as it needs to contain information about the _entire_ target system. Which, for distributed systems, is a costly operation. So using this as a path checker is not making certain array vendors happy.
From working on one of these arrays, many of the path state decisions often require the same set of information to respond correctly to test unit ready. Other OS's including both Windows and ESX, once they determine that the array supports ALUA, use RTPG rather than TUR as the periodic path checker.
Quoted 14 lines
However: as you might have followed the discussion about scsi multipathing, maybe we can turn this around to profit both sides:
Can't we move ALUA state handling into the generic code, leaving the device handler only to deal with explicit ALUA?
That way we can intercept sense codes and send RTPG only when required, such that the ALUA state of the scsi device is correct.
And the the ALUA path checker just has to evaluate the information in sysfs.
_And_ scsi multipathing can use that information, too.
Quoted 1 line
Hmm?
I think that there are likely ways to re-imagine the checker and prioritizer separation that would have benefit. This patch wasn't really about going in "as is" but to start that sort of thought.
In general multipath-tools is "good enough" to support the ALUA story from my perspective as a target vendor. There are some improvements that would likely simplify things.
The Linux kernel, in my view, should take these two changes to complete its ALUA story:
https://marc.info/?l=linux-scsi&m=177189801604075&w=2
https://marc.info/?l=linux-scsi&m=176530648031820&w=2
It would be great if someone could review those two changes.
Quoted 9 lines
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
[email protected] +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Thanks,
Brian
Signature · 3 lines
Brian Bunker PURE Storage, Inc. [email protected]
Re: [PATCH 0/1] pr: add alua path checker
In reply to Brian BunkerQuoted 6 lines · Brian Bunker
For ALUA-capable storage arrays, multipath-tools currently uses TUR (Test Unit Ready) as the default path checker while a separate prioritizer determines ALUA state. When sysfs provides ALUA state (the common case), no RTPG command is needed for priority. However, if sysfs is unavailable or
detect_priois off, the prioritizer issues RTPG, resulting in two SCSI commands per path check cycle: TUR + RTPG.
prio/checker changes should be documented in multipath/multipath.conf.5.in
Thanks.