ceph: implement new-style ENOSPC handling in kcephfs
Subsystems (2)
v2 cover ceph: implement new-style ENOSPC handling in kcephfs
v2: teach libcephfs how to hold on to requests until the right map epoch appears, instead of delaying cap handling in the cephfs layer.
This patchset is an updated version of the patch series originally done by John Spray and posted here:
http://www.spinics.net/lists/ceph-devel/msg21257.html
The patchset has undergone a number of changes since the original submission, and now more closely resembles the approach taken by the userland cephfs code. When it sees a new epoch barrier, it communicates that to libcephfs which then uses the normal request target pausing mechanism to hold them until the right map comes in.
With this, xfstests seems to work as well as before, and we get timely -ENOSPC returns under these conditions with O_DIRECT writes. I still need to plumb in a way to throttle the dirtying of new pages in buffered I/O when we are getting errors during writeback. Still, I figure this is a good place to pause to make sure that things look ok in the set before I implement that part.
Jeff Layton (5):
libceph: add an epoch_barrier field to struct ceph_osd_client
libceph: rename and export have_pool_full
ceph: register map callback to handle ENOSPC conditions
ceph: handle epoch barriers in cap messages
libceph: allow requests to return immediately on full conditions if
caller wishes
John Spray (1):
libceph: add ceph_osdc_complete_writes
fs/ceph/addr.c | 14 ++++-- fs/ceph/caps.c | 17 +++++-- fs/ceph/file.c | 8 +-- fs/ceph/mds_client.c | 38 +++++++++++++++ fs/ceph/mds_client.h | 7 ++- include/linux/ceph/osd_client.h | 15 ++++++ include/linux/ceph/rados.h | 1 + net/ceph/osd_client.c | 105 +++++++++++++++++++++++++++++++++++----- 8 files changed, 180 insertions(+), 25 deletions(-)
Signature · 1 line
2.9.3
PATCH v2 1/6 libceph: add an epoch_barrier field to struct ceph_osd_client
Cephfs can get cap update requests that contain a new epoch barrier in them. When that happens we want to pause all OSD traffic until the right map epoch arrives. Add a way for the upper layers to set the epoch_barrier in ceph_osd_client and fix libceph to pause requests until it has the right map epoch.
Add an epoch_barrier field to ceph_osd_client that is protected by the osdc->lock rwsem. When the barrier is set, and the current OSD map epoch is below that, pause the request target when submitting the request or when revisiting it.
If we get a new map, compare the new epoch against the barrier before kicking requests and request another map if the map epoch is still lower than the one we want.
--- include/linux/ceph/osd_client.h | 2 ++ net/ceph/osd_client.c | 42 +++++++++++++++++++++++++++++++++-------- 2 files changed, 36 insertions(+), 8 deletions(-)
include/linux/ceph/osd_client.h +2 -0
@@ -269,6 +269,7 @@ struct ceph_osd_client {struct rb_root osds;/* osds */ struct list_head osd_lru;/* idle osds */ spinlock_t osd_lru_lock; +u32 epoch_barrier;struct ceph_osd homeless_osd;atomic64_t last_tid;/* tid of last request */ u64 last_linger_id;
@@ -307,6 +308,7 @@ extern void ceph_osdc_handle_reply(struct ceph_osd_client *osdc, struct ceph_msg *msg);extern void ceph_osdc_handle_map (struct ceph_osd_client *osdc,struct ceph_msg *msg); +void ceph_osdc_update_epoch_barrier (struct ceph_osd_client *osdc,u32 eb);extern void osd_req_op_init (struct ceph_osd_request *osd_req,unsigned int which,u16 opcode,u32 flags);
net/ceph/osd_client.c +34 -8
84 lines not on this page · raw message
Signature · 1 line
2.9.3
PATCH v2 2/6 libceph: add ceph_osdc_complete_writes
In reply to Jeff LaytonFrom: John Spray <[email protected]>
When a Ceph volume hits capacity, a flag is set in the OSD map to indicate that, and a new map is sprayed around the cluster. When the cephfs client sees that, we want it to shut down any OSD writes that are in-progress with an -ENOSPC error as they'll just hang otherwise.
Add a callback to the osdc that gets called on map updates and a way for upper layers to register that callback.
[ jlayton: code style cleanup and adaptation to new osd msg handling ]
--- include/linux/ceph/osd_client.h | 12 ++++++++++ net/ceph/osd_client.c | 50 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+)
include/linux/ceph/osd_client.h +12 -0
@@ -21,6 +21,7 @@ struct ceph_osd_client;/* * completion callback for async writepages */ +typedef void (*ceph_osdc_map_callback_t )(struct ceph_osd_client *,void *);typedef void (*ceph_osdc_callback_t )(struct ceph_osd_request *);typedef void (*ceph_osdc_unsafe_callback_t )(struct ceph_osd_request *,bool );
@@ -290,6 +291,9 @@ struct ceph_osd_client {struct ceph_msgpool msgpool_op_reply;struct workqueue_struct *notify_wq; + +ceph_osdc_map_callback_t map_cb; +void *map_p; };static inline bool ceph_osdmap_flag (struct ceph_osd_client *osdc,int flag)
@@ -393,6 +397,7 @@ extern void ceph_osdc_put_request(struct ceph_osd_request *req);extern int ceph_osdc_start_request (struct ceph_osd_client *osdc,struct ceph_osd_request *req,bool nofail); +extern u32 ceph_osdc_complete_writes (struct ceph_osd_client *osdc,int r);extern void ceph_osdc_cancel_request (struct ceph_osd_request *req);extern int ceph_osdc_wait_request (struct ceph_osd_client *osdc,struct ceph_osd_request *req);
@@ -459,5 +464,12 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc,struct ceph_object_locator *oloc, struct ceph_watch_item **watchers, u32 *num_watchers); + +static inline void ceph_osdc_register_map_cb (struct ceph_osd_client *osdc, +ceph_osdc_map_callback_t cb,void *data) +{ + osdc->map_cb = cb; + osdc->map_p = data; +}#endif
net/ceph/osd_client.c +50 -0
74 lines not on this page · raw message
Signature · 1 line
2.9.3
PATCH v2 3/6 libceph: rename and export have_pool_full
In reply to Jeff LaytonCephfs needs to be able to call this as well. Rename have_pool_full to ceph_osdc_have_pool_full, and export it.
--- include/linux/ceph/osd_client.h | 1 + net/ceph/osd_client.c | 7 ++++--- 2 files changed, 5 insertions(+), 3 deletions(-)
include/linux/ceph/osd_client.h +1 -0
@@ -313,6 +313,7 @@ extern void ceph_osdc_handle_reply(struct ceph_osd_client *osdc,extern void ceph_osdc_handle_map (struct ceph_osd_client *osdc,struct ceph_msg *msg);void ceph_osdc_update_epoch_barrier (struct ceph_osd_client *osdc,u32 eb); +extern bool ceph_osdc_have_pool_full (struct ceph_osd_client *osdc);extern void osd_req_op_init (struct ceph_osd_request *osd_req,unsigned int which,u16 opcode,u32 flags);
net/ceph/osd_client.c +4 -3
@@ -1259,7 +1259,7 @@ static bool __pool_full(struct ceph_pg_pool_info *pi)return pi->flags &CEPH_POOL_FLAG_FULL ; } -static bool have_pool_full (struct ceph_osd_client *osdc) +bool ceph_osdc_have_pool_full (struct ceph_osd_client *osdc) {struct rb_node *n;
@@ -1273,6 +1273,7 @@ static bool have_pool_full(struct ceph_osd_client *osdc)return false; } +EXPORT_SYMBOL (ceph_osdc_have_pool_full);static bool pool_full (struct ceph_osd_client *osdc,s64 pool_id) {
@@ -3271,7 +3272,7 @@ void ceph_osdc_handle_map(struct ceph_osd_client *osdc, struct ceph_msg *msg) was_pauserd =ceph_osdmap_flag (osdc,CEPH_OSDMAP_PAUSERD ); was_pausewr =ceph_osdmap_flag (osdc,CEPH_OSDMAP_PAUSEWR ) ||ceph_osdmap_flag (osdc,CEPH_OSDMAP_FULL ) || -have_pool_full (osdc); +ceph_osdc_have_pool_full (osdc);/* incremental maps */ ceph_decode_32_safe (&p, end, nr_maps, bad);
@@ -3335,7 +3336,7 @@ void ceph_osdc_handle_map(struct ceph_osd_client *osdc, struct ceph_msg *msg) pauserd =ceph_osdmap_flag (osdc,CEPH_OSDMAP_PAUSERD ); pausewr =ceph_osdmap_flag (osdc,CEPH_OSDMAP_PAUSEWR ) ||ceph_osdmap_flag (osdc,CEPH_OSDMAP_FULL ) || -have_pool_full (osdc); +ceph_osdc_have_pool_full (osdc);if (was_pauserd || was_pausewr || pauserd || pausewr || (osdc->epoch_barrier && osdc->osdmap->epoch < osdc->epoch_barrier))maybe_request_map (osdc);
Signature · 1 line
2.9.3
PATCH v2 4/6 ceph: register map callback to handle ENOSPC conditions
In reply to Jeff Layton--- fs/ceph/mds_client.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
fs/ceph/mds_client.c +18 -0
@@ -433,6 +433,21 @@ static int __verify_registered_session(struct ceph_mds_client *mdsc,return 0 ; } +static void handle_osd_map (struct ceph_osd_client *osdc,void *p) +{ +u32 cancelled_epoch =0 ; + +lockdep_assert_held (&osdc->lock); + +if ((osdc->osdmap->flags &CEPH_OSDMAP_FULL ) || +ceph_osdc_have_pool_full (osdc)) + cancelled_epoch =ceph_osdc_complete_writes (osdc, -ENOSPC ); + +if (cancelled_epoch) + osdc->epoch_barrier =max (cancelled_epoch +1 , + osdc->epoch_barrier); +} +/* * create+register a new session for given mds. * called under mdsc->mutex.
@@ -3486,6 +3501,9 @@ int ceph_mdsc_init(struct ceph_fs_client *fsc)ceph_caps_init (mdsc);ceph_adjust_min_caps (mdsc, fsc->min_caps); +ceph_osdc_register_map_cb (&fsc->client->osdc, + handle_osd_map, (void *)mdsc); +init_rwsem (&mdsc->pool_perm_rwsem); mdsc->pool_perm_tree =RB_ROOT ;
Signature · 1 line
2.9.3
PATCH v2 5/6 ceph: handle epoch barriers in cap messages
In reply to Jeff LaytonHave the client store and update the osdc epoch_barrier when a cap message comes in with one. This allows clients to inform servers that their released caps may not be used until a particular OSD map epoch.
--- fs/ceph/caps.c | 17 +++++++++++++---- fs/ceph/mds_client.c | 20 ++++++++++++++++++++ fs/ceph/mds_client.h | 7 +++++-- 3 files changed, 38 insertions(+), 6 deletions(-)
fs/ceph/caps.c +13 -4
@@ -1015,6 +1015,7 @@ static int send_cap_msg(struct cap_msg_args *arg)void *p;size_t extra_len;struct timespec zerotime = {0 }; +struct ceph_osd_client *osdc = &arg->session->s_mdsc->fsc->client->osdc;dout ("send_cap_msg %s %llx %llx caps %s wanted %s dirty %s" " seq %u/%u tid %llu/%llu mseq %u follows %lld size %llu/%llu"
@@ -1077,7 +1078,9 @@ static int send_cap_msg(struct cap_msg_args *arg)/* inline data size */ ceph_encode_32 (&p,0 );/* osd_epoch_barrier (version 5) */ -ceph_encode_32 (&p,0 ); +down_read (&osdc->lock); +ceph_encode_32 (&p, osdc->epoch_barrier); +up_read (&osdc->lock);/* oldest_flush_tid (version 6) */ ceph_encode_64 (&p, arg->oldest_flush_tid);
@@ -3635,13 +3638,19 @@ void ceph_handle_caps(struct ceph_mds_session *session, p += inline_len; } +if (le16_to_cpu (msg->hdr.version) >=5 ) { +struct ceph_osd_client *osdc = &mdsc->fsc->client->osdc; +u32 epoch_barrier; + +ceph_decode_32_safe (&p, end, epoch_barrier, bad); +ceph_osdc_update_epoch_barrier (osdc, epoch_barrier); + } +if (le16_to_cpu (msg->hdr.version) >=8 ) {u64 flush_tid;u32 caller_uid, caller_gid; -u32 osd_epoch_barrier;u32 pool_ns_len; -/* version >= 5 */ -ceph_decode_32_safe (&p, end, osd_epoch_barrier, bad); +/* version >= 6 */ ceph_decode_64_safe (&p, end, flush_tid, bad);/* version >= 7 */
fs/ceph/mds_client.c +20 -0
@@ -1565,9 +1565,15 @@ void ceph_send_cap_releases(struct ceph_mds_client *mdsc,struct ceph_msg *msg =NULL ;struct ceph_mds_cap_release *head;struct ceph_mds_cap_item *item; +struct ceph_osd_client *osdc = &mdsc->fsc->client->osdc;struct ceph_cap *cap;LIST_HEAD (tmp_list);int num_cap_releases; +__le32 barrier, *cap_barrier; + +down_read (&osdc->lock); + barrier =cpu_to_le32 (osdc->epoch_barrier); +up_read (&osdc->lock);spin_lock (&session->s_cap_lock); again:
@@ -1585,7 +1591,11 @@ void ceph_send_cap_releases(struct ceph_mds_client *mdsc, head = msg->front.iov_base; head->num =cpu_to_le32 (0 ); msg->front.iov_len =sizeof (*head); + + msg->hdr.version =cpu_to_le16 (2 ); + msg->hdr.compat_version =cpu_to_le16 (1 ); } + cap = list_first_entry(&tmp_list, struct ceph_cap, session_caps);list_del (&cap->session_caps);
@@ -1603,6 +1613,11 @@ void ceph_send_cap_releases(struct ceph_mds_client *mdsc,ceph_put_cap (mdsc, cap);if (le32_to_cpu (head->num) ==CEPH_CAPS_PER_RELEASE ) { +// Append cap_barrier field + cap_barrier = msg->front.iov_base + msg->front.iov_len; + *cap_barrier = barrier; + msg->front.iov_len +=sizeof (*cap_barrier); + msg->hdr.front_len =cpu_to_le32 (msg->front.iov_len);dout ("send_cap_releases mds%d %p\n" , session->s_mds, msg);ceph_con_send (&session->s_con, msg);
@@ -1618,6 +1633,11 @@ void ceph_send_cap_releases(struct ceph_mds_client *mdsc,spin_unlock (&session->s_cap_lock);if (msg) { +// Append cap_barrier field + cap_barrier = msg->front.iov_base + msg->front.iov_len; + *cap_barrier = barrier; + msg->front.iov_len +=sizeof (*cap_barrier); + msg->hdr.front_len =cpu_to_le32 (msg->front.iov_len);dout ("send_cap_releases mds%d %p\n" , session->s_mds, msg);ceph_con_send (&session->s_con, msg);
fs/ceph/mds_client.h +5 -2
@@ -104,10 +104,13 @@ struct ceph_mds_reply_info_parsed {/* * cap releases are batched and sent to the MDS en masse. +* +* Account for per-message overhead of mds_cap_release header +* and __le32 for osd epoch barrier trailing field. */ -#define CEPH_CAPS_PER_RELEASE ((PAGE_SIZE - \ +#define CEPH_CAPS_PER_RELEASE ((PAGE_SIZE - sizeof(u32) - \ sizeof(struct ceph_mds_cap_release)) / \ - sizeof(struct ceph_mds_cap_item)) + sizeof(struct ceph_mds_cap_item))/*
Signature · 1 line
2.9.3
PATCH v2 6/6 libceph: allow requests to return immediately on full conditions if caller wishes
In reply to Jeff LaytonRight now, cephfs will cancel any in-flight OSD write operations when a new map comes in that shows the OSD or pool as full, but nothing prevents new requests from stalling out after that point.
If the caller knows that it will want an immediate error return instead of blocking on a full or at-quota error condition then allow it to set a flag to request that behavior. Cephfs write requests will always set that flag.
--- fs/ceph/addr.c | 14 +++++++++----- fs/ceph/file.c | 8 +++++--- include/linux/ceph/rados.h | 1 + net/ceph/osd_client.c | 6 ++++++ 4 files changed, 21 insertions(+), 8 deletions(-)
fs/ceph/addr.c +9 -5
@@ -1019,7 +1019,8 @@ static int ceph_writepages_start(struct address_space *mapping, offset, &len,0 , num_ops,CEPH_OSD_OP_WRITE ,CEPH_OSD_FLAG_WRITE | -CEPH_OSD_FLAG_ONDISK , +CEPH_OSD_FLAG_ONDISK | +CEPH_OSD_FLAG_FULL_CANCEL , snapc, truncate_seq, truncate_size, false);if (IS_ERR (req)) {
@@ -1030,7 +1031,8 @@ static int ceph_writepages_start(struct address_space *mapping, CEPH_OSD_SLAB_OPS),CEPH_OSD_OP_WRITE ,CEPH_OSD_FLAG_WRITE | -CEPH_OSD_FLAG_ONDISK , +CEPH_OSD_FLAG_ONDISK | +CEPH_OSD_FLAG_FULL_CANCEL , snapc, truncate_seq, truncate_size, true);BUG_ON (IS_ERR (req));
@@ -1681,7 +1683,9 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) req =ceph_osdc_new_request (&fsc->client->osdc, &ci->i_layout,ceph_vino (inode),0 , &len,0 ,1 ,CEPH_OSD_OP_CREATE , -CEPH_OSD_FLAG_ONDISK |CEPH_OSD_FLAG_WRITE , +CEPH_OSD_FLAG_ONDISK | +CEPH_OSD_FLAG_WRITE | +CEPH_OSD_FLAG_FULL_CANCEL ,NULL ,0 ,0 , false);if (IS_ERR (req)) { err =PTR_ERR (req);
@@ -1699,7 +1703,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) req =ceph_osdc_new_request (&fsc->client->osdc, &ci->i_layout,ceph_vino (inode),0 , &len,1 ,3 ,CEPH_OSD_OP_WRITE , -CEPH_OSD_FLAG_ONDISK |CEPH_OSD_FLAG_WRITE , +CEPH_OSD_FLAG_ONDISK |CEPH_OSD_FLAG_WRITE |CEPH_OSD_FLAG_FULL_CANCEL ,NULL , ci->i_truncate_seq, ci->i_truncate_size, false);if (IS_ERR (req)) {
@@ -1872,7 +1876,7 @@ static int __ceph_pool_perm_get(struct ceph_inode_info *ci, goto out_unlock; } - wr_req->r_flags =CEPH_OSD_FLAG_WRITE |CEPH_OSD_FLAG_ACK ; + wr_req->r_flags =CEPH_OSD_FLAG_WRITE |CEPH_OSD_FLAG_ACK |CEPH_OSD_FLAG_FULL_CANCEL ;osd_req_op_init (wr_req,0 ,CEPH_OSD_OP_CREATE ,CEPH_OSD_OP_FLAG_EXCL );ceph_oloc_copy (&wr_req->r_base_oloc, &rd_req->r_base_oloc);ceph_oid_copy (&wr_req->r_base_oid, &rd_req->r_base_oid);
fs/ceph/file.c +5 -3
@@ -692,7 +692,7 @@ static void ceph_aio_retry_work(struct work_struct *work) req->r_flags =CEPH_OSD_FLAG_ORDERSNAP |CEPH_OSD_FLAG_ONDISK | -CEPH_OSD_FLAG_WRITE ; +CEPH_OSD_FLAG_WRITE |CEPH_OSD_FLAG_FULL_CANCEL ;ceph_oloc_copy (&req->r_base_oloc, &orig_req->r_base_oloc);ceph_oid_copy (&req->r_base_oid, &orig_req->r_base_oid);
@@ -849,7 +849,7 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter, flags =CEPH_OSD_FLAG_ORDERSNAP |CEPH_OSD_FLAG_ONDISK | -CEPH_OSD_FLAG_WRITE ; +CEPH_OSD_FLAG_WRITE |CEPH_OSD_FLAG_FULL_CANCEL ; }else { flags =CEPH_OSD_FLAG_READ ; }
@@ -1051,6 +1051,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, flags =CEPH_OSD_FLAG_ORDERSNAP |CEPH_OSD_FLAG_ONDISK |CEPH_OSD_FLAG_WRITE | +CEPH_OSD_FLAG_FULL_CANCEL |CEPH_OSD_FLAG_ACK ;while ((len =iov_iter_count (from)) >0 ) {
@@ -1549,7 +1550,8 @@ static int ceph_zero_partial_object(struct inode *inode, offset, length,0 ,1 , op,CEPH_OSD_FLAG_WRITE | -CEPH_OSD_FLAG_ONDISK , +CEPH_OSD_FLAG_ONDISK | +CEPH_OSD_FLAG_FULL_CANCEL , NULL, 0, 0, false);if (IS_ERR (req)) { ret =PTR_ERR (req);
include/linux/ceph/rados.h +1 -0
@@ -401,6 +401,7 @@ enum {CEPH_OSD_FLAG_KNOWN_REDIR =0x400000 ,/* redirect bit is authoritative */ CEPH_OSD_FLAG_FULL_TRY =0x800000 ,/* try op despite full flag */ CEPH_OSD_FLAG_FULL_FORCE =0x1000000 ,/* force op despite full flag */ + CEPH_OSD_FLAG_FULL_CANCEL = 0x2000000, /* cancel operation on full flag */ }; enum {
net/ceph/osd_client.c +6 -0
@@ -50,6 +50,7 @@ static void link_linger(struct ceph_osd *osd, struct ceph_osd_linger_request *lreq);static void unlink_linger (struct ceph_osd *osd,struct ceph_osd_linger_request *lreq); +static void complete_request (struct ceph_osd_request *req,int err);#if 1 static inline bool rwsem_is_wrlocked (struct rw_semaphore *sem)
@@ -1643,6 +1644,7 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)enum calc_target_result ct_res;bool need_send = false;bool promoted = false; +int ret =0 ;WARN_ON (req->r_tid || req->r_got_reply);dout ("%s req %p wrlocked %d\n" , __func__, req, wrlocked);
@@ -1683,6 +1685,8 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)pr_warn_ratelimited ("FULL or reached pool quota\n" ); req->r_t.paused = true;maybe_request_map (osdc); +if (req->r_flags &CEPH_OSD_FLAG_FULL_CANCEL ) + ret = -ENOSPC ; }else if (!osd_homeless (osd)) { need_send = true; }else {
@@ -1699,6 +1703,8 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)link_request (osd, req);if (need_send)send_request (req); +else if (ret) +complete_request (req, ret);mutex_unlock (&osd->lock);if (ct_res ==CALC_TARGET_POOL_DNE )
Signature · 1 line
2.9.3
Ilya Dryomov
maintainerRe: [PATCH v2 6/6] libceph: allow requests to return immediately on full conditions if caller wishes
Quoted 115 lines · Jeff Layton
Right now, cephfs will cancel any in-flight OSD write operations when a new map comes in that shows the OSD or pool as full, but nothing prevents new requests from stalling out after that point.
If the caller knows that it will want an immediate error return instead of blocking on a full or at-quota error condition then allow it to set a flag to request that behavior. Cephfs write requests will always set that flag.
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/addr.c | 14 +++++++++----- fs/ceph/file.c | 8 +++++--- include/linux/ceph/rados.h | 1 + net/ceph/osd_client.c | 6 ++++++ 4 files changed, 21 insertions(+), 8 deletions(-)fs/ceph/addr.c +9
-5@@ -1019,7 +1019,8 @@ static int ceph_writepages_start(struct address_space *mapping, offset, &len,0 , num_ops,CEPH_OSD_OP_WRITE ,CEPH_OSD_FLAG_WRITE | -CEPH_OSD_FLAG_ONDISK , +CEPH_OSD_FLAG_ONDISK | +CEPH_OSD_FLAG_FULL_CANCEL , snapc, truncate_seq, truncate_size, false);if (IS_ERR (req)) {@@ -1030,7 +1031,8 @@ static int ceph_writepages_start(struct address_space *mapping, CEPH_OSD_SLAB_OPS),CEPH_OSD_OP_WRITE ,CEPH_OSD_FLAG_WRITE | -CEPH_OSD_FLAG_ONDISK , +CEPH_OSD_FLAG_ONDISK | +CEPH_OSD_FLAG_FULL_CANCEL , snapc, truncate_seq, truncate_size, true);BUG_ON (IS_ERR (req));@@ -1681,7 +1683,9 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) req =ceph_osdc_new_request (&fsc->client->osdc, &ci->i_layout,ceph_vino (inode),0 , &len,0 ,1 ,CEPH_OSD_OP_CREATE , -CEPH_OSD_FLAG_ONDISK |CEPH_OSD_FLAG_WRITE , +CEPH_OSD_FLAG_ONDISK | +CEPH_OSD_FLAG_WRITE | +CEPH_OSD_FLAG_FULL_CANCEL ,NULL ,0 ,0 , false);if (IS_ERR (req)) { err =PTR_ERR (req);@@ -1699,7 +1703,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) req =ceph_osdc_new_request (&fsc->client->osdc, &ci->i_layout,ceph_vino (inode),0 , &len,1 ,3 ,CEPH_OSD_OP_WRITE , -CEPH_OSD_FLAG_ONDISK |CEPH_OSD_FLAG_WRITE , +CEPH_OSD_FLAG_ONDISK |CEPH_OSD_FLAG_WRITE |CEPH_OSD_FLAG_FULL_CANCEL ,NULL , ci->i_truncate_seq, ci->i_truncate_size, false);if (IS_ERR (req)) {@@ -1872,7 +1876,7 @@ static int __ceph_pool_perm_get(struct ceph_inode_info *ci, goto out_unlock; } - wr_req->r_flags =CEPH_OSD_FLAG_WRITE |CEPH_OSD_FLAG_ACK ; + wr_req->r_flags =CEPH_OSD_FLAG_WRITE |CEPH_OSD_FLAG_ACK |CEPH_OSD_FLAG_FULL_CANCEL ;osd_req_op_init (wr_req,0 ,CEPH_OSD_OP_CREATE ,CEPH_OSD_OP_FLAG_EXCL );ceph_oloc_copy (&wr_req->r_base_oloc, &rd_req->r_base_oloc);ceph_oid_copy (&wr_req->r_base_oid, &rd_req->r_base_oid);fs/ceph/file.c +5
-3@@ -692,7 +692,7 @@ static void ceph_aio_retry_work(struct work_struct *work) req->r_flags =CEPH_OSD_FLAG_ORDERSNAP |CEPH_OSD_FLAG_ONDISK | -CEPH_OSD_FLAG_WRITE ; +CEPH_OSD_FLAG_WRITE |CEPH_OSD_FLAG_FULL_CANCEL ;ceph_oloc_copy (&req->r_base_oloc, &orig_req->r_base_oloc);ceph_oid_copy (&req->r_base_oid, &orig_req->r_base_oid);@@ -849,7 +849,7 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter, flags =CEPH_OSD_FLAG_ORDERSNAP |CEPH_OSD_FLAG_ONDISK | -CEPH_OSD_FLAG_WRITE ; +CEPH_OSD_FLAG_WRITE |CEPH_OSD_FLAG_FULL_CANCEL ; }else { flags =CEPH_OSD_FLAG_READ ; }@@ -1051,6 +1051,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, flags =CEPH_OSD_FLAG_ORDERSNAP |CEPH_OSD_FLAG_ONDISK |CEPH_OSD_FLAG_WRITE | +CEPH_OSD_FLAG_FULL_CANCEL |CEPH_OSD_FLAG_ACK ;while ((len =iov_iter_count (from)) >0 ) {@@ -1549,7 +1550,8 @@ static int ceph_zero_partial_object(struct inode *inode, offset, length,0 ,1 , op,CEPH_OSD_FLAG_WRITE | -CEPH_OSD_FLAG_ONDISK , +CEPH_OSD_FLAG_ONDISK | +CEPH_OSD_FLAG_FULL_CANCEL , NULL, 0, 0, false);if (IS_ERR (req)) { ret =PTR_ERR (req);include/linux/ceph/rados.h +0
-0
include/linux/ceph/rados.h enum { · line 404
Quoted 4 lines
include/linux/ceph/rados.h +1
-0@@ -401,6 +401,7 @@ enum {CEPH_OSD_FLAG_KNOWN_REDIR =0x400000 ,/* redirect bit is authoritative */ CEPH_OSD_FLAG_FULL_TRY =0x800000 ,/* try op despite full flag */ CEPH_OSD_FLAG_FULL_FORCE =0x1000000 ,/* force op despite full flag */ + CEPH_OSD_FLAG_FULL_CANCEL = 0x2000000, /* cancel operation on full flag */
Is this a new flag? This is the wire protocol and I don't see it in ceph.git.
I'll look at epoch_barrier and callback stuff later.
Thanks,
Ilya
Re: [PATCH v2 6/6] libceph: allow requests to return immediately on full conditions if caller wishes
Quoted 124 lines · Ilya Dryomov
Quoted 115 lines · Jeff Layton
Right now, cephfs will cancel any in-flight OSD write operations when a new map comes in that shows the OSD or pool as full, but nothing prevents new requests from stalling out after that point.
If the caller knows that it will want an immediate error return instead of blocking on a full or at-quota error condition then allow it to set a flag to request that behavior. Cephfs write requests will always set that flag.
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/addr.c | 14 +++++++++----- fs/ceph/file.c | 8 +++++--- include/linux/ceph/rados.h | 1 + net/ceph/osd_client.c | 6 ++++++ 4 files changed, 21 insertions(+), 8 deletions(-)fs/ceph/addr.c +9
-5@@ -1019,7 +1019,8 @@ static int ceph_writepages_start(struct address_space *mapping, offset, &len,0 , num_ops,CEPH_OSD_OP_WRITE ,CEPH_OSD_FLAG_WRITE | -CEPH_OSD_FLAG_ONDISK , +CEPH_OSD_FLAG_ONDISK | +CEPH_OSD_FLAG_FULL_CANCEL , snapc, truncate_seq, truncate_size, false);if (IS_ERR (req)) {@@ -1030,7 +1031,8 @@ static int ceph_writepages_start(struct address_space *mapping, CEPH_OSD_SLAB_OPS),CEPH_OSD_OP_WRITE ,CEPH_OSD_FLAG_WRITE | -CEPH_OSD_FLAG_ONDISK , +CEPH_OSD_FLAG_ONDISK | +CEPH_OSD_FLAG_FULL_CANCEL , snapc, truncate_seq, truncate_size, true);BUG_ON (IS_ERR (req));@@ -1681,7 +1683,9 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) req =ceph_osdc_new_request (&fsc->client->osdc, &ci->i_layout,ceph_vino (inode),0 , &len,0 ,1 ,CEPH_OSD_OP_CREATE , -CEPH_OSD_FLAG_ONDISK |CEPH_OSD_FLAG_WRITE , +CEPH_OSD_FLAG_ONDISK | +CEPH_OSD_FLAG_WRITE | +CEPH_OSD_FLAG_FULL_CANCEL ,NULL ,0 ,0 , false);if (IS_ERR (req)) { err =PTR_ERR (req);@@ -1699,7 +1703,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) req =ceph_osdc_new_request (&fsc->client->osdc, &ci->i_layout,ceph_vino (inode),0 , &len,1 ,3 ,CEPH_OSD_OP_WRITE , -CEPH_OSD_FLAG_ONDISK |CEPH_OSD_FLAG_WRITE , +CEPH_OSD_FLAG_ONDISK |CEPH_OSD_FLAG_WRITE |CEPH_OSD_FLAG_FULL_CANCEL ,NULL , ci->i_truncate_seq, ci->i_truncate_size, false);if (IS_ERR (req)) {@@ -1872,7 +1876,7 @@ static int __ceph_pool_perm_get(struct ceph_inode_info *ci, goto out_unlock; } - wr_req->r_flags =CEPH_OSD_FLAG_WRITE |CEPH_OSD_FLAG_ACK ; + wr_req->r_flags =CEPH_OSD_FLAG_WRITE |CEPH_OSD_FLAG_ACK |CEPH_OSD_FLAG_FULL_CANCEL ;osd_req_op_init (wr_req,0 ,CEPH_OSD_OP_CREATE ,CEPH_OSD_OP_FLAG_EXCL );ceph_oloc_copy (&wr_req->r_base_oloc, &rd_req->r_base_oloc);ceph_oid_copy (&wr_req->r_base_oid, &rd_req->r_base_oid);fs/ceph/file.c +5
-3@@ -692,7 +692,7 @@ static void ceph_aio_retry_work(struct work_struct *work) req->r_flags =CEPH_OSD_FLAG_ORDERSNAP |CEPH_OSD_FLAG_ONDISK | -CEPH_OSD_FLAG_WRITE ; +CEPH_OSD_FLAG_WRITE |CEPH_OSD_FLAG_FULL_CANCEL ;ceph_oloc_copy (&req->r_base_oloc, &orig_req->r_base_oloc);ceph_oid_copy (&req->r_base_oid, &orig_req->r_base_oid);@@ -849,7 +849,7 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter, flags =CEPH_OSD_FLAG_ORDERSNAP |CEPH_OSD_FLAG_ONDISK | -CEPH_OSD_FLAG_WRITE ; +CEPH_OSD_FLAG_WRITE |CEPH_OSD_FLAG_FULL_CANCEL ; }else { flags =CEPH_OSD_FLAG_READ ; }@@ -1051,6 +1051,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, flags =CEPH_OSD_FLAG_ORDERSNAP |CEPH_OSD_FLAG_ONDISK |CEPH_OSD_FLAG_WRITE | +CEPH_OSD_FLAG_FULL_CANCEL |CEPH_OSD_FLAG_ACK ;while ((len =iov_iter_count (from)) >0 ) {@@ -1549,7 +1550,8 @@ static int ceph_zero_partial_object(struct inode *inode, offset, length,0 ,1 , op,CEPH_OSD_FLAG_WRITE | -CEPH_OSD_FLAG_ONDISK , +CEPH_OSD_FLAG_ONDISK | +CEPH_OSD_FLAG_FULL_CANCEL , NULL, 0, 0, false);if (IS_ERR (req)) { ret =PTR_ERR (req);include/linux/ceph/rados.h +0
-0include/linux/ceph/rados.h enum { · line 404
Quoted 4 lines
include/linux/ceph/rados.h +1
-0@@ -401,6 +401,7 @@ enum {CEPH_OSD_FLAG_KNOWN_REDIR =0x400000 ,/* redirect bit is authoritative */ CEPH_OSD_FLAG_FULL_TRY =0x800000 ,/* try op despite full flag */ CEPH_OSD_FLAG_FULL_FORCE =0x1000000 ,/* force op despite full flag */ + CEPH_OSD_FLAG_FULL_CANCEL = 0x2000000, /* cancel operation on full flag */Is this a new flag? This is the wire protocol and I don't see it in ceph.git.
Quoted 4 lines
I'll look at
epoch_barrierand callback stuff later.Thanks,
Oof, ok. I thought those were kernel-internal flags. I missed that they get encoded onto the wire.
Yeah, this is probably the wrong place to pass that flag in then. What we really want is to pass this along to the request submission code, but there is no need to pass this to the server.
I'll look at the code to see if there's a more suitable place for this flag. Worst case, I'll just add a new bool to ceph_osd_request for this.
Thanks,
Signature · 1 line
Jeff Layton <[email protected]>
Re: [PATCH v2 6/6] libceph: allow requests to return immediately on full conditions if caller wishes
In reply to Ilya DryomovOn Mon, 2017-02-06 at 15:09 +0100, Ilya Dryomov wrote:
[...]
Quoted 13 lines
include/linux/ceph/rados.h enum { · line 404
Quoted 9 lines
include/linux/ceph/rados.h +1
-0@@ -401,6 +401,7 @@ enum {CEPH_OSD_FLAG_KNOWN_REDIR =0x400000 ,/* redirect bit is authoritative */ CEPH_OSD_FLAG_FULL_TRY =0x800000 ,/* try op despite full flag */ CEPH_OSD_FLAG_FULL_FORCE =0x1000000 ,/* force op despite full flag */ + CEPH_OSD_FLAG_FULL_CANCEL = 0x2000000, /* cancel operation on full flag */Is this a new flag? This is the wire protocol and I don't see it in ceph.git.
Quoted 5 lines
I'll look at
epoch_barrierand callback stuff later.Thanks,
Ilya
Here's a respun version of the last patch in the set. This should avoid adding an on the wire flag. I just added a new bool and changed the code to set and look at that to indicate the desire for an immediate error return in this case. Compiles but is otherwise untested. I'll give it a go in a bit.
-----------------------------8<------------------------------
libceph: allow requests to return immediately on full
conditions if caller wishes
Right now, cephfs will cancel any in-flight OSD write operations when a new map comes in that shows the OSD or pool as full, but nothing prevents new requests from stalling out after that point.
If the caller knows that it will want an immediate error return instead of blocking on a full or at-quota error condition then allow it to set a flag to request that behavior. Cephfs write requests will always set that flag.
--- fs/ceph/addr.c | 4 ++++ fs/ceph/file.c | 4 ++++ include/linux/ceph/osd_client.h | 1 + net/ceph/osd_client.c | 6 ++++++ 4 files changed, 15 insertions(+)
fs/ceph/addr.c +4 -0
@@ -1040,6 +1040,7 @@ static int ceph_writepages_start(struct address_space *mapping, req->r_callback = writepages_finish; req->r_inode = inode; + req->r_enospc_on_full = true;/* Format the osd request message and submit the write */ len =0 ;
@@ -1689,6 +1690,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; err =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!err) err =ceph_osdc_wait_request (&fsc->client->osdc, req);
@@ -1732,6 +1734,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; err =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!err) err =ceph_osdc_wait_request (&fsc->client->osdc, req);
@@ -1893,6 +1896,7 @@ static int __ceph_pool_perm_get(struct ceph_inode_info *ci, err =ceph_osdc_start_request (&fsc->client->osdc, rd_req, false); wr_req->r_mtime = ci->vfs_inode.i_mtime; + wr_req->r_enospc_on_full = true; err2 =ceph_osdc_start_request (&fsc->client->osdc, wr_req, false);if (!err)
fs/ceph/file.c +4 -0
@@ -714,6 +714,7 @@ static void ceph_aio_retry_work(struct work_struct *work) req->r_callback = ceph_aio_complete_req; req->r_inode = inode; req->r_priv = aio_req; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (req->r_osdc, req, false); out:
@@ -912,6 +913,7 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,osd_req_op_init (req,1 ,CEPH_OSD_OP_STARTSYNC ,0 ); req->r_mtime = mtime; + req->r_enospc_on_full = true; } osd_req_op_extent_osd_data_pages(req, 0, pages, len, start,
@@ -1105,6 +1107,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, false, true); req->r_mtime = mtime; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!ret) ret =ceph_osdc_wait_request (&fsc->client->osdc, req);
@@ -1557,6 +1560,7 @@ static int ceph_zero_partial_object(struct inode *inode, } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!ret) { ret =ceph_osdc_wait_request (&fsc->client->osdc, req);
include/linux/ceph/osd_client.h +1 -0
@@ -172,6 +172,7 @@ struct ceph_osd_request {int r_result;bool r_got_reply; +bool r_enospc_on_full;/* return ENOSPC when full */ struct ceph_osd_client *r_osdc;struct kref r_kref;
net/ceph/osd_client.c +6 -0
@@ -50,6 +50,7 @@ static void link_linger(struct ceph_osd *osd, struct ceph_osd_linger_request *lreq);static void unlink_linger (struct ceph_osd *osd,struct ceph_osd_linger_request *lreq); +static void complete_request (struct ceph_osd_request *req,int err);#if 1 static inline bool rwsem_is_wrlocked (struct rw_semaphore *sem)
@@ -1643,6 +1644,7 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)enum calc_target_result ct_res;bool need_send = false;bool promoted = false; +int ret =0 ;WARN_ON (req->r_tid || req->r_got_reply);dout ("%s req %p wrlocked %d\n" , __func__, req, wrlocked);
@@ -1683,6 +1685,8 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)pr_warn_ratelimited ("FULL or reached pool quota\n" ); req->r_t.paused = true;maybe_request_map (osdc); +if (req->r_enospc_on_full) + ret = -ENOSPC ; }else if (!osd_homeless (osd)) { need_send = true; }else {
@@ -1699,6 +1703,8 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)link_request (osd, req);if (need_send)send_request (req); +else if (ret) +complete_request (req, ret);mutex_unlock (&osd->lock);if (ct_res ==CALC_TARGET_POOL_DNE )
Signature · 1 line
2.9.3
Ilya Dryomov
maintainerRe: [PATCH v2 6/6] libceph: allow requests to return immediately on full conditions if caller wishes
Quoted 165 lines · Jeff Layton
On Mon, 2017-02-06 at 15:09 +0100, Ilya Dryomov wrote:
[...]
Quoted 13 lines
include/linux/ceph/rados.h enum { · line 404
Quoted 9 lines
include/linux/ceph/rados.h +1
-0@@ -401,6 +401,7 @@ enum {CEPH_OSD_FLAG_KNOWN_REDIR =0x400000 ,/* redirect bit is authoritative */ CEPH_OSD_FLAG_FULL_TRY =0x800000 ,/* try op despite full flag */ CEPH_OSD_FLAG_FULL_FORCE =0x1000000 ,/* force op despite full flag */ + CEPH_OSD_FLAG_FULL_CANCEL = 0x2000000, /* cancel operation on full flag */Is this a new flag? This is the wire protocol and I don't see it in ceph.git.
Quoted 5 lines
I'll look at
epoch_barrierand callback stuff later.Thanks,
Ilya
Here's a respun version of the last patch in the set. This should avoid adding an on the wire flag. I just added a new bool and changed the code to set and look at that to indicate the desire for an immediate error return in this case. Compiles but is otherwise untested. I'll give it a go in a bit.
-----------------------------8<------------------------------
libceph: allow requests to return immediately on full
conditions if caller wishesRight now, cephfs will cancel any in-flight OSD write operations when a new map comes in that shows the OSD or pool as full, but nothing prevents new requests from stalling out after that point.
If the caller knows that it will want an immediate error return instead of blocking on a full or at-quota error condition then allow it to set a flag to request that behavior. Cephfs write requests will always set that flag.
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/addr.c | 4 ++++ fs/ceph/file.c | 4 ++++ include/linux/ceph/osd_client.h | 1 + net/ceph/osd_client.c | 6 ++++++ 4 files changed, 15 insertions(+)fs/ceph/addr.c +4
-0@@ -1040,6 +1040,7 @@ static int ceph_writepages_start(struct address_space *mapping, req->r_callback = writepages_finish; req->r_inode = inode; + req->r_enospc_on_full = true;/* Format the osd request message and submit the write */ len =0 ;@@ -1689,6 +1690,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; err =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!err) err =ceph_osdc_wait_request (&fsc->client->osdc, req);@@ -1732,6 +1734,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; err =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!err) err =ceph_osdc_wait_request (&fsc->client->osdc, req);@@ -1893,6 +1896,7 @@ static int __ceph_pool_perm_get(struct ceph_inode_info *ci, err =ceph_osdc_start_request (&fsc->client->osdc, rd_req, false); wr_req->r_mtime = ci->vfs_inode.i_mtime; + wr_req->r_enospc_on_full = true; err2 =ceph_osdc_start_request (&fsc->client->osdc, wr_req, false);if (!err)fs/ceph/file.c +4
-0@@ -714,6 +714,7 @@ static void ceph_aio_retry_work(struct work_struct *work) req->r_callback = ceph_aio_complete_req; req->r_inode = inode; req->r_priv = aio_req; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (req->r_osdc, req, false); out:@@ -912,6 +913,7 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,osd_req_op_init (req,1 ,CEPH_OSD_OP_STARTSYNC ,0 ); req->r_mtime = mtime; + req->r_enospc_on_full = true; } osd_req_op_extent_osd_data_pages(req, 0, pages, len, start,@@ -1105,6 +1107,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, false, true); req->r_mtime = mtime; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!ret) ret =ceph_osdc_wait_request (&fsc->client->osdc, req);@@ -1557,6 +1560,7 @@ static int ceph_zero_partial_object(struct inode *inode, } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!ret) { ret =ceph_osdc_wait_request (&fsc->client->osdc, req);include/linux/ceph/osd_client.h +1
-0@@ -172,6 +172,7 @@ struct ceph_osd_request {int r_result;bool r_got_reply; +bool r_enospc_on_full;/* return ENOSPC when full */ struct ceph_osd_client *r_osdc;struct kref r_kref;net/ceph/osd_client.c +4
-0@@ -50,6 +50,7 @@ static void link_linger(struct ceph_osd *osd, struct ceph_osd_linger_request *lreq);static void unlink_linger (struct ceph_osd *osd,struct ceph_osd_linger_request *lreq); +static void complete_request (struct ceph_osd_request *req,int err);#if 1 static inline bool rwsem_is_wrlocked (struct rw_semaphore *sem)@@ -1643,6 +1644,7 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)enum calc_target_result ct_res;bool need_send = false;bool promoted = false; +int ret =0 ;WARN_ON (req->r_tid || req->r_got_reply);dout ("%s req %p wrlocked %d\n" , __func__, req, wrlocked);@@ -1683,6 +1685,8 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)pr_warn_ratelimited ("FULL or reached pool quota\n" ); req->r_t.paused = true;maybe_request_map (osdc); +if (req->r_enospc_on_full) + ret = -ENOSPC ; }else if (!osd_homeless (osd)) { need_send = true; }else {
net/ceph/osd_client.c static void __submit_request(struct ceph_osd_request *req, bool wrlocked) · line 1707
Quoted 5 lines
net/ceph/osd_client.c +2
-0@@ -1699,6 +1703,8 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)link_request (osd, req);if (need_send)send_request (req); +else if (ret) +complete_request (req, ret);
How is this handled in the userspace client? I don't see a similar check in Objecter.
Thanks,
Ilya
Re: [PATCH v2 6/6] libceph: allow requests to return immediately on full conditions if caller wishes
Quoted 175 lines · Ilya Dryomov
Quoted 165 lines · Jeff Layton
On Mon, 2017-02-06 at 15:09 +0100, Ilya Dryomov wrote:
[...]
Quoted 13 lines
include/linux/ceph/rados.h enum { · line 404
Quoted 9 lines
include/linux/ceph/rados.h +1
-0@@ -401,6 +401,7 @@ enum {CEPH_OSD_FLAG_KNOWN_REDIR =0x400000 ,/* redirect bit is authoritative */ CEPH_OSD_FLAG_FULL_TRY =0x800000 ,/* try op despite full flag */ CEPH_OSD_FLAG_FULL_FORCE =0x1000000 ,/* force op despite full flag */ + CEPH_OSD_FLAG_FULL_CANCEL = 0x2000000, /* cancel operation on full flag */Is this a new flag? This is the wire protocol and I don't see it in ceph.git.
Quoted 5 lines
I'll look at
epoch_barrierand callback stuff later.Thanks,
Ilya
Here's a respun version of the last patch in the set. This should avoid adding an on the wire flag. I just added a new bool and changed the code to set and look at that to indicate the desire for an immediate error return in this case. Compiles but is otherwise untested. I'll give it a go in a bit.
-----------------------------8<------------------------------
libceph: allow requests to return immediately on full
conditions if caller wishesRight now, cephfs will cancel any in-flight OSD write operations when a new map comes in that shows the OSD or pool as full, but nothing prevents new requests from stalling out after that point.
If the caller knows that it will want an immediate error return instead of blocking on a full or at-quota error condition then allow it to set a flag to request that behavior. Cephfs write requests will always set that flag.
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/addr.c | 4 ++++ fs/ceph/file.c | 4 ++++ include/linux/ceph/osd_client.h | 1 + net/ceph/osd_client.c | 6 ++++++ 4 files changed, 15 insertions(+)fs/ceph/addr.c +4
-0@@ -1040,6 +1040,7 @@ static int ceph_writepages_start(struct address_space *mapping, req->r_callback = writepages_finish; req->r_inode = inode; + req->r_enospc_on_full = true;/* Format the osd request message and submit the write */ len =0 ;@@ -1689,6 +1690,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; err =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!err) err =ceph_osdc_wait_request (&fsc->client->osdc, req);@@ -1732,6 +1734,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; err =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!err) err =ceph_osdc_wait_request (&fsc->client->osdc, req);@@ -1893,6 +1896,7 @@ static int __ceph_pool_perm_get(struct ceph_inode_info *ci, err =ceph_osdc_start_request (&fsc->client->osdc, rd_req, false); wr_req->r_mtime = ci->vfs_inode.i_mtime; + wr_req->r_enospc_on_full = true; err2 =ceph_osdc_start_request (&fsc->client->osdc, wr_req, false);if (!err)fs/ceph/file.c +4
-0@@ -714,6 +714,7 @@ static void ceph_aio_retry_work(struct work_struct *work) req->r_callback = ceph_aio_complete_req; req->r_inode = inode; req->r_priv = aio_req; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (req->r_osdc, req, false); out:@@ -912,6 +913,7 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,osd_req_op_init (req,1 ,CEPH_OSD_OP_STARTSYNC ,0 ); req->r_mtime = mtime; + req->r_enospc_on_full = true; } osd_req_op_extent_osd_data_pages(req, 0, pages, len, start,@@ -1105,6 +1107,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, false, true); req->r_mtime = mtime; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!ret) ret =ceph_osdc_wait_request (&fsc->client->osdc, req);@@ -1557,6 +1560,7 @@ static int ceph_zero_partial_object(struct inode *inode, } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!ret) { ret =ceph_osdc_wait_request (&fsc->client->osdc, req);include/linux/ceph/osd_client.h +1
-0@@ -172,6 +172,7 @@ struct ceph_osd_request {int r_result;bool r_got_reply; +bool r_enospc_on_full;/* return ENOSPC when full */ struct ceph_osd_client *r_osdc;struct kref r_kref;net/ceph/osd_client.c +4
-0@@ -50,6 +50,7 @@ static void link_linger(struct ceph_osd *osd, struct ceph_osd_linger_request *lreq);static void unlink_linger (struct ceph_osd *osd,struct ceph_osd_linger_request *lreq); +static void complete_request (struct ceph_osd_request *req,int err);#if 1 static inline bool rwsem_is_wrlocked (struct rw_semaphore *sem)@@ -1643,6 +1644,7 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)enum calc_target_result ct_res;bool need_send = false;bool promoted = false; +int ret =0 ;WARN_ON (req->r_tid || req->r_got_reply);dout ("%s req %p wrlocked %d\n" , __func__, req, wrlocked);@@ -1683,6 +1685,8 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)pr_warn_ratelimited ("FULL or reached pool quota\n" ); req->r_t.paused = true;maybe_request_map (osdc); +if (req->r_enospc_on_full) + ret = -ENOSPC ; }else if (!osd_homeless (osd)) { need_send = true; }else {net/ceph/osd_client.c static void __submit_request(struct ceph_osd_request *req, bool wrlocked) · line 1707
Quoted 5 lines
net/ceph/osd_client.c +2
-0@@ -1699,6 +1703,8 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)link_request (osd, req);if (need_send)send_request (req); +else if (ret) +complete_request (req, ret);How is this handled in the userspace client? I don't see a similar check in Objecter.
Quoted 3 lines
Thanks,
Ilya
It seems to be handled at a much higher layer in libcephfs. In _write(), for instance, we have:
if (objecter->osdmap_pool_full(in->layout.pool_id)) { return -ENOSPC;
}
...and there's also some EDQUOT handling a little below there as well.
I don't think we can reasonably follow that model in the kernel client though. The way it's done in userland seems to require the big client mutex be held over large swaths of the code, and that's not the case in the kernel client (thankfully).
Signature · 1 line
Jeff Layton <[email protected]>
Ilya Dryomov
maintainerRe: [PATCH v2 6/6] libceph: allow requests to return immediately on full conditions if caller wishes
Quoted 187 lines · Jeff Layton
Quoted 175 lines · Ilya Dryomov
Quoted 165 lines · Jeff Layton
On Mon, 2017-02-06 at 15:09 +0100, Ilya Dryomov wrote:
[...]
Quoted 13 lines
include/linux/ceph/rados.h enum { · line 404
Quoted 9 lines
include/linux/ceph/rados.h +1
-0@@ -401,6 +401,7 @@ enum {CEPH_OSD_FLAG_KNOWN_REDIR =0x400000 ,/* redirect bit is authoritative */ CEPH_OSD_FLAG_FULL_TRY =0x800000 ,/* try op despite full flag */ CEPH_OSD_FLAG_FULL_FORCE =0x1000000 ,/* force op despite full flag */ + CEPH_OSD_FLAG_FULL_CANCEL = 0x2000000, /* cancel operation on full flag */Is this a new flag? This is the wire protocol and I don't see it in ceph.git.
Quoted 5 lines
I'll look at
epoch_barrierand callback stuff later.Thanks,
Ilya
Here's a respun version of the last patch in the set. This should avoid adding an on the wire flag. I just added a new bool and changed the code to set and look at that to indicate the desire for an immediate error return in this case. Compiles but is otherwise untested. I'll give it a go in a bit.
-----------------------------8<------------------------------
libceph: allow requests to return immediately on full
conditions if caller wishesRight now, cephfs will cancel any in-flight OSD write operations when a new map comes in that shows the OSD or pool as full, but nothing prevents new requests from stalling out after that point.
If the caller knows that it will want an immediate error return instead of blocking on a full or at-quota error condition then allow it to set a flag to request that behavior. Cephfs write requests will always set that flag.
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/addr.c | 4 ++++ fs/ceph/file.c | 4 ++++ include/linux/ceph/osd_client.h | 1 + net/ceph/osd_client.c | 6 ++++++ 4 files changed, 15 insertions(+)fs/ceph/addr.c +4
-0@@ -1040,6 +1040,7 @@ static int ceph_writepages_start(struct address_space *mapping, req->r_callback = writepages_finish; req->r_inode = inode; + req->r_enospc_on_full = true;/* Format the osd request message and submit the write */ len =0 ;@@ -1689,6 +1690,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; err =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!err) err =ceph_osdc_wait_request (&fsc->client->osdc, req);@@ -1732,6 +1734,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; err =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!err) err =ceph_osdc_wait_request (&fsc->client->osdc, req);@@ -1893,6 +1896,7 @@ static int __ceph_pool_perm_get(struct ceph_inode_info *ci, err =ceph_osdc_start_request (&fsc->client->osdc, rd_req, false); wr_req->r_mtime = ci->vfs_inode.i_mtime; + wr_req->r_enospc_on_full = true; err2 =ceph_osdc_start_request (&fsc->client->osdc, wr_req, false);if (!err)fs/ceph/file.c +4
-0@@ -714,6 +714,7 @@ static void ceph_aio_retry_work(struct work_struct *work) req->r_callback = ceph_aio_complete_req; req->r_inode = inode; req->r_priv = aio_req; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (req->r_osdc, req, false); out:@@ -912,6 +913,7 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,osd_req_op_init (req,1 ,CEPH_OSD_OP_STARTSYNC ,0 ); req->r_mtime = mtime; + req->r_enospc_on_full = true; } osd_req_op_extent_osd_data_pages(req, 0, pages, len, start,@@ -1105,6 +1107,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, false, true); req->r_mtime = mtime; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!ret) ret =ceph_osdc_wait_request (&fsc->client->osdc, req);@@ -1557,6 +1560,7 @@ static int ceph_zero_partial_object(struct inode *inode, } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!ret) { ret =ceph_osdc_wait_request (&fsc->client->osdc, req);include/linux/ceph/osd_client.h +1
-0@@ -172,6 +172,7 @@ struct ceph_osd_request {int r_result;bool r_got_reply; +bool r_enospc_on_full;/* return ENOSPC when full */ struct ceph_osd_client *r_osdc;struct kref r_kref;net/ceph/osd_client.c +4
-0@@ -50,6 +50,7 @@ static void link_linger(struct ceph_osd *osd, struct ceph_osd_linger_request *lreq);static void unlink_linger (struct ceph_osd *osd,struct ceph_osd_linger_request *lreq); +static void complete_request (struct ceph_osd_request *req,int err);#if 1 static inline bool rwsem_is_wrlocked (struct rw_semaphore *sem)@@ -1643,6 +1644,7 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)enum calc_target_result ct_res;bool need_send = false;bool promoted = false; +int ret =0 ;WARN_ON (req->r_tid || req->r_got_reply);dout ("%s req %p wrlocked %d\n" , __func__, req, wrlocked);@@ -1683,6 +1685,8 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)pr_warn_ratelimited ("FULL or reached pool quota\n" ); req->r_t.paused = true;maybe_request_map (osdc); +if (req->r_enospc_on_full) + ret = -ENOSPC ; }else if (!osd_homeless (osd)) { need_send = true; }else {net/ceph/osd_client.c static void __submit_request(struct ceph_osd_request *req, bool wrlocked) · line 1707
Quoted 5 lines
net/ceph/osd_client.c +2
-0@@ -1699,6 +1703,8 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)link_request (osd, req);if (need_send)send_request (req); +else if (ret) +complete_request (req, ret);How is this handled in the userspace client? I don't see a similar check in Objecter.
Quoted 3 lines
Thanks,
Ilya
It seems to be handled at a much higher layer in libcephfs. In
_write(), for instance, we have:if (objecter->osdmap_pool_full(in->layout.pool_id)) { return -ENOSPC; }
Quoted 6 lines
...and there's also some EDQUOT handling a little below there as well.
I don't think we can reasonably follow that model in the kernel client though. The way it's done in userland seems to require the big client mutex be held over large swaths of the code, and that's not the case in the kernel client (thankfully).
All it's doing is checking a flag under a lock; the same lock is taken in ceph_osdc_start_request(). Why would exporting
bool ceph_osdc_pool_full(osdc, pool_id)
{
bool ret;
down_read(&osdc->lock);
ret = ceph_osdmap_flag(osdc, CEPH_OSDMAP_FULL) ||
pool_full(osdc, pool_id);
up_read(&osdc->lock);
return ret;
}not work? IOW why special case request handling code if you know in advance that the request is a goner?
Thanks,
Ilya
Re: [PATCH v2 6/6] libceph: allow requests to return immediately on full conditions if caller wishes
Quoted 209 lines · Ilya Dryomov
Quoted 187 lines · Jeff Layton
Quoted 175 lines · Ilya Dryomov
Quoted 165 lines · Jeff Layton
On Mon, 2017-02-06 at 15:09 +0100, Ilya Dryomov wrote:
[...]
Quoted 13 lines
include/linux/ceph/rados.h enum { · line 404
Quoted 9 lines
include/linux/ceph/rados.h +1
-0@@ -401,6 +401,7 @@ enum {CEPH_OSD_FLAG_KNOWN_REDIR =0x400000 ,/* redirect bit is authoritative */ CEPH_OSD_FLAG_FULL_TRY =0x800000 ,/* try op despite full flag */ CEPH_OSD_FLAG_FULL_FORCE =0x1000000 ,/* force op despite full flag */ + CEPH_OSD_FLAG_FULL_CANCEL = 0x2000000, /* cancel operation on full flag */Is this a new flag? This is the wire protocol and I don't see it in ceph.git.
Quoted 5 lines
I'll look at
epoch_barrierand callback stuff later.Thanks,
Ilya
Here's a respun version of the last patch in the set. This should avoid adding an on the wire flag. I just added a new bool and changed the code to set and look at that to indicate the desire for an immediate error return in this case. Compiles but is otherwise untested. I'll give it a go in a bit.
-----------------------------8<------------------------------
libceph: allow requests to return immediately on full
conditions if caller wishesRight now, cephfs will cancel any in-flight OSD write operations when a new map comes in that shows the OSD or pool as full, but nothing prevents new requests from stalling out after that point.
If the caller knows that it will want an immediate error return instead of blocking on a full or at-quota error condition then allow it to set a flag to request that behavior. Cephfs write requests will always set that flag.
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/addr.c | 4 ++++ fs/ceph/file.c | 4 ++++ include/linux/ceph/osd_client.h | 1 + net/ceph/osd_client.c | 6 ++++++ 4 files changed, 15 insertions(+)fs/ceph/addr.c +4
-0@@ -1040,6 +1040,7 @@ static int ceph_writepages_start(struct address_space *mapping, req->r_callback = writepages_finish; req->r_inode = inode; + req->r_enospc_on_full = true;/* Format the osd request message and submit the write */ len =0 ;@@ -1689,6 +1690,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; err =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!err) err =ceph_osdc_wait_request (&fsc->client->osdc, req);@@ -1732,6 +1734,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; err =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!err) err =ceph_osdc_wait_request (&fsc->client->osdc, req);@@ -1893,6 +1896,7 @@ static int __ceph_pool_perm_get(struct ceph_inode_info *ci, err =ceph_osdc_start_request (&fsc->client->osdc, rd_req, false); wr_req->r_mtime = ci->vfs_inode.i_mtime; + wr_req->r_enospc_on_full = true; err2 =ceph_osdc_start_request (&fsc->client->osdc, wr_req, false);if (!err)fs/ceph/file.c +4
-0@@ -714,6 +714,7 @@ static void ceph_aio_retry_work(struct work_struct *work) req->r_callback = ceph_aio_complete_req; req->r_inode = inode; req->r_priv = aio_req; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (req->r_osdc, req, false); out:@@ -912,6 +913,7 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,osd_req_op_init (req,1 ,CEPH_OSD_OP_STARTSYNC ,0 ); req->r_mtime = mtime; + req->r_enospc_on_full = true; } osd_req_op_extent_osd_data_pages(req, 0, pages, len, start,@@ -1105,6 +1107,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, false, true); req->r_mtime = mtime; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!ret) ret =ceph_osdc_wait_request (&fsc->client->osdc, req);@@ -1557,6 +1560,7 @@ static int ceph_zero_partial_object(struct inode *inode, } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!ret) { ret =ceph_osdc_wait_request (&fsc->client->osdc, req);include/linux/ceph/osd_client.h +1
-0@@ -172,6 +172,7 @@ struct ceph_osd_request {int r_result;bool r_got_reply; +bool r_enospc_on_full;/* return ENOSPC when full */ struct ceph_osd_client *r_osdc;struct kref r_kref;net/ceph/osd_client.c +4
-0@@ -50,6 +50,7 @@ static void link_linger(struct ceph_osd *osd, struct ceph_osd_linger_request *lreq);static void unlink_linger (struct ceph_osd *osd,struct ceph_osd_linger_request *lreq); +static void complete_request (struct ceph_osd_request *req,int err);#if 1 static inline bool rwsem_is_wrlocked (struct rw_semaphore *sem)@@ -1643,6 +1644,7 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)enum calc_target_result ct_res;bool need_send = false;bool promoted = false; +int ret =0 ;WARN_ON (req->r_tid || req->r_got_reply);dout ("%s req %p wrlocked %d\n" , __func__, req, wrlocked);@@ -1683,6 +1685,8 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)pr_warn_ratelimited ("FULL or reached pool quota\n" ); req->r_t.paused = true;maybe_request_map (osdc); +if (req->r_enospc_on_full) + ret = -ENOSPC ; }else if (!osd_homeless (osd)) { need_send = true; }else {net/ceph/osd_client.c static void __submit_request(struct ceph_osd_request *req, bool wrlocked) · line 1707
Quoted 5 lines
net/ceph/osd_client.c +2
-0@@ -1699,6 +1703,8 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)link_request (osd, req);if (need_send)send_request (req); +else if (ret) +complete_request (req, ret);How is this handled in the userspace client? I don't see a similar check in Objecter.
Quoted 3 lines
Thanks,
Ilya
It seems to be handled at a much higher layer in libcephfs. In
_write(), for instance, we have:if (objecter->osdmap_pool_full(in->layout.pool_id)) { return -ENOSPC; }Quoted 6 lines
...and there's also some EDQUOT handling a little below there as well.
I don't think we can reasonably follow that model in the kernel client though. The way it's done in userland seems to require the big client mutex be held over large swaths of the code, and that's not the case in the kernel client (thankfully).
All it's doing is checking a flag under a lock; the same lock is taken in
ceph_osdc_start_request(). Why would exportingbool
ceph_osdc_pool_full(osdc, pool_id){ bool ret; down_read(&osdc->lock); ret = ceph_osdmap_flag(osdc, CEPH_OSDMAP_FULL) || pool_full(osdc, pool_id); up_read(&osdc->lock); return ret; }
Quoted 3 lines
not work? IOW why special case request handling code if you know in advance that the request is a goner?
You can't just call that before calling submit_request because once you call up_read there, the map could have changed and the map_cb already run. At that point we can't rely on the callback cancelling the now hung request.
You could call that function after ceph_osdc_start_request, but before we end up waiting on it (or whatever), but then we'd need to export complete_request as well so we could cancel it. The other downside there is that we end up with even more rwsem thrashing. We already end up taking it for write in order to submit the thing, so having to take it for read again afterward is not ideal.
We could remedy that by pushing the rwsem handling up into cephfs (basically, export submit_request too), but that smells more like a layering violation to me. The full flag is part of the osdmap, so it doesn't seem wrong to have libcephfs handle it, and that's quite a bit simpler and more efficient than trying to manage that from the cephfs layer.
Signature · 1 line
Jeff Layton <[email protected]>
Re: [PATCH v2 6/6] libceph: allow requests to return immediately on full conditions if caller wishes
Quoted 227 lines · Jeff Layton
Quoted 209 lines · Ilya Dryomov
Quoted 187 lines · Jeff Layton
Quoted 175 lines · Ilya Dryomov
Quoted 165 lines · Jeff Layton
On Mon, 2017-02-06 at 15:09 +0100, Ilya Dryomov wrote:
[...]
Quoted 13 lines
include/linux/ceph/rados.h enum { · line 404
Quoted 9 lines
include/linux/ceph/rados.h +1
-0@@ -401,6 +401,7 @@ enum {CEPH_OSD_FLAG_KNOWN_REDIR =0x400000 ,/* redirect bit is authoritative */ CEPH_OSD_FLAG_FULL_TRY =0x800000 ,/* try op despite full flag */ CEPH_OSD_FLAG_FULL_FORCE =0x1000000 ,/* force op despite full flag */ + CEPH_OSD_FLAG_FULL_CANCEL = 0x2000000, /* cancel operation on full flag */Is this a new flag? This is the wire protocol and I don't see it in ceph.git.
Quoted 5 lines
I'll look at
epoch_barrierand callback stuff later.Thanks,
Ilya
Here's a respun version of the last patch in the set. This should avoid adding an on the wire flag. I just added a new bool and changed the code to set and look at that to indicate the desire for an immediate error return in this case. Compiles but is otherwise untested. I'll give it a go in a bit.
-----------------------------8<------------------------------
libceph: allow requests to return immediately on full
conditions if caller wishesRight now, cephfs will cancel any in-flight OSD write operations when a new map comes in that shows the OSD or pool as full, but nothing prevents new requests from stalling out after that point.
If the caller knows that it will want an immediate error return instead of blocking on a full or at-quota error condition then allow it to set a flag to request that behavior. Cephfs write requests will always set that flag.
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/addr.c | 4 ++++ fs/ceph/file.c | 4 ++++ include/linux/ceph/osd_client.h | 1 + net/ceph/osd_client.c | 6 ++++++ 4 files changed, 15 insertions(+)fs/ceph/addr.c +4
-0@@ -1040,6 +1040,7 @@ static int ceph_writepages_start(struct address_space *mapping, req->r_callback = writepages_finish; req->r_inode = inode; + req->r_enospc_on_full = true;/* Format the osd request message and submit the write */ len =0 ;@@ -1689,6 +1690,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; err =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!err) err =ceph_osdc_wait_request (&fsc->client->osdc, req);@@ -1732,6 +1734,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; err =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!err) err =ceph_osdc_wait_request (&fsc->client->osdc, req);@@ -1893,6 +1896,7 @@ static int __ceph_pool_perm_get(struct ceph_inode_info *ci, err =ceph_osdc_start_request (&fsc->client->osdc, rd_req, false); wr_req->r_mtime = ci->vfs_inode.i_mtime; + wr_req->r_enospc_on_full = true; err2 =ceph_osdc_start_request (&fsc->client->osdc, wr_req, false);if (!err)fs/ceph/file.c +4
-0@@ -714,6 +714,7 @@ static void ceph_aio_retry_work(struct work_struct *work) req->r_callback = ceph_aio_complete_req; req->r_inode = inode; req->r_priv = aio_req; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (req->r_osdc, req, false); out:@@ -912,6 +913,7 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,osd_req_op_init (req,1 ,CEPH_OSD_OP_STARTSYNC ,0 ); req->r_mtime = mtime; + req->r_enospc_on_full = true; } osd_req_op_extent_osd_data_pages(req, 0, pages, len, start,@@ -1105,6 +1107,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, false, true); req->r_mtime = mtime; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!ret) ret =ceph_osdc_wait_request (&fsc->client->osdc, req);@@ -1557,6 +1560,7 @@ static int ceph_zero_partial_object(struct inode *inode, } req->r_mtime = inode->i_mtime; + req->r_enospc_on_full = true; ret =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!ret) { ret =ceph_osdc_wait_request (&fsc->client->osdc, req);include/linux/ceph/osd_client.h +1
-0@@ -172,6 +172,7 @@ struct ceph_osd_request {int r_result;bool r_got_reply; +bool r_enospc_on_full;/* return ENOSPC when full */ struct ceph_osd_client *r_osdc;struct kref r_kref;net/ceph/osd_client.c +4
-0@@ -50,6 +50,7 @@ static void link_linger(struct ceph_osd *osd, struct ceph_osd_linger_request *lreq);static void unlink_linger (struct ceph_osd *osd,struct ceph_osd_linger_request *lreq); +static void complete_request (struct ceph_osd_request *req,int err);#if 1 static inline bool rwsem_is_wrlocked (struct rw_semaphore *sem)@@ -1643,6 +1644,7 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)enum calc_target_result ct_res;bool need_send = false;bool promoted = false; +int ret =0 ;WARN_ON (req->r_tid || req->r_got_reply);dout ("%s req %p wrlocked %d\n" , __func__, req, wrlocked);@@ -1683,6 +1685,8 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)pr_warn_ratelimited ("FULL or reached pool quota\n" ); req->r_t.paused = true;maybe_request_map (osdc); +if (req->r_enospc_on_full) + ret = -ENOSPC ; }else if (!osd_homeless (osd)) { need_send = true; }else {net/ceph/osd_client.c static void __submit_request(struct ceph_osd_request *req, bool wrlocked) · line 1707
Quoted 5 lines
net/ceph/osd_client.c +2
-0@@ -1699,6 +1703,8 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)link_request (osd, req);if (need_send)send_request (req); +else if (ret) +complete_request (req, ret);How is this handled in the userspace client? I don't see a similar check in Objecter.
Quoted 3 lines
Thanks,
Ilya
It seems to be handled at a much higher layer in libcephfs. In
_write(), for instance, we have:if (objecter->osdmap_pool_full(in->layout.pool_id)) { return -ENOSPC; }Quoted 6 lines
...and there's also some EDQUOT handling a little below there as well.
I don't think we can reasonably follow that model in the kernel client though. The way it's done in userland seems to require the big client mutex be held over large swaths of the code, and that's not the case in the kernel client (thankfully).
All it's doing is checking a flag under a lock; the same lock is taken in
ceph_osdc_start_request(). Why would exportingbool
ceph_osdc_pool_full(osdc, pool_id){ bool ret; down_read(&osdc->lock); ret = ceph_osdmap_flag(osdc, CEPH_OSDMAP_FULL) || pool_full(osdc, pool_id); up_read(&osdc->lock); return ret; }Quoted 3 lines
not work? IOW why special case request handling code if you know in advance that the request is a goner?
You can't just call that before calling
submit_requestbecause once you callup_readthere, the map could have changed and themap_cbalready run. At that point we can't rely on the callback cancelling the now hung request.You could call that function after
ceph_osdc_start_request, but before we end up waiting on it (or whatever), but then we'd need to exportcomplete_requestas well so we could cancel it. The other downside there is that we end up with even more rwsem thrashing. We already end up taking it for write in order to submit the thing, so having to take it for read again afterward is not ideal.
Quoted 6 lines
We could remedy that by pushing the rwsem handling up into cephfs (basically, export
submit_requesttoo), but that smells more like a layering violation to me. The full flag is part of the osdmap, so it doesn't seem wrong to have libcephfs handle it, and that's quite a bit simpler and more efficient than trying to manage that from the cephfs layer.
That said...I think it is better to use this new flag in a more coherent fashion in the complete_writes callback. I've done some cleanup and slimdown of the set today, so you may want to hold off reviewing this just yet, and wait until I repost.
Thanks,
Signature · 1 line
Jeff Layton <[email protected]>