Home
My Posts

Archive
Login
Signup

Recent Posts

Free subdomains

Want your own xyz.kpaste.net sub-domain for your community? Just type the address into your browser address bar (foo.xyz.kpaste.net is also supported).

Suggestions/Bugs?

Email us at . Try to include as much info as possible.

About

Pastebin is a tool for collaborative debugging or editing.

 

msnfs41client: Patch for |FSCTL_SET_ZERO_DATA|, NFSv4.2 ALLOCATE+DEALLOCATE, workaround for Linux nfsd SEEK bug, tests+misc, 2025-02-14
Posted by Anonymous on Fri 14th Feb 2025 16:11
raw | new post

Submit a correction or amendment below (click here to make a fresh posting)
After submitting an amendment, you'll be able to view the differences between the old and new posts easily.

Syntax highlighting: None Bash C C++ CSS Diff HTML mIRC Script Perl PHP Python Smarty XML ---------------------------- ABAP Actionscript ADA Apache Log AppleScript APT sources.list ASM ASP AutoIT Bash C C# C++ C++ (with QT) CSS Diff HTML INI (Config Files) Java Javascript Lua Make mIRC Script MySQL NSIS Perl PHP Python Q(uick)BASIC robots Ruby Ruby on Rails SDLBasic Smarty SQL TCL Text VB.NET Visual BASIC Visual Fox Pro Visual Prolog Windows Registry Files XML Xorg.conf

To highlight particular lines, prefix each line with {%HIGHLIGHT}
From 0a8bb72df688fe68fab6e207d8afc5fd87f3309b Mon Sep 17 00:00:00 2001 From: Roland Mainz <roland.mainz@nrubsig.org> Date: Fri, 14 Feb 2025 11:46:46 +0100 Subject: [PATCH 1/3] daemon,tests: Add workaround for Linux nfsd NFSv4.2 SEEK |NFS4ERR_NXIO| bug Add a workaround for the Linux NFSv4.2 nfsd SEEK bug which returns |NFS4ERR_NXIO| if it cannot find a data block in a sparse file (i.e. a file which consists only of a single hole and no data), The NFSv4.2 RFC says in https://datatracker.ietf.org/doc/html/rfc7862#section-15.11.3 that "... If the server cannot find a corresponding sa_what, then the status will still be NFS4_OK, but sr_eof would be TRUE. ...", but the Linux nfsd SEEK does not behave that way. Question is... which offset should a conforming NFSv4.2 SEEK_DATA return if there is no data block (i.e. sparse file which only consists of one hole) ? Also add tests "hole-only" sparse files and sparse files with a hole at the end. Signed-off-by: Cedric Blancher <cedric.blancher@gmail.com> --- daemon/daemon_debug.c | 4 + daemon/fsctl.c | 28 +++++++ tests/sparsefiles/testsparsefile1.ksh | 112 +++++++++++++++++++++++--- 3 files changed, 132 insertions(+), 12 deletions(-) diff --git a/daemon/daemon_debug.c b/daemon/daemon_debug.c index fa0f90c..ab7f567 100644 --- a/daemon/daemon_debug.c +++ b/daemon/daemon_debug.c @@ -1183,6 +1183,10 @@ void debug_list_sparsefile_holes(nfs41_open_state *state) * https://datatracker.ietf.org/doc/html/rfc7862#section-15.11.3 * states "If the sa_offset is beyond the end of the file, then * SEEK MUST return NFS4ERR_NXIO." + * + * Question is... which offset should a conforming NFSv4.2 + * SEEK_DATA return if there is no data block (i.e. sparse + * file which only consists of one hole) ? */ #define LINUX_NFSD_SEEK_NXIO_BUG_WORKAROUND 1 diff --git a/daemon/fsctl.c b/daemon/fsctl.c index 61b9dca..421909d 100644 --- a/daemon/fsctl.c +++ b/daemon/fsctl.c @@ -97,6 +97,34 @@ int query_sparsefile_datasections(nfs41_open_state *state, NFS4_CONTENT_DATA, &data_seek_sr_eof, &data_seek_sr_offset); + + /* + * 1. Note that Linux returns |NFS4ERR_NXIO| if it cannot find + * a data block, but + * https://datatracker.ietf.org/doc/html/rfc7862#section-15.11.3 + * says "... If the server cannot find a corresponding sa_what, + * then the status will still be NFS4_OK, but sr_eof would be + * TRUE. ..." + * 2. NFSv4.2 spec bug: + * https://datatracker.ietf.org/doc/html/rfc7862#section-11.2 + * section "SEEK" does not list |NFS4ERR_NXIO| as valid error + * for SEEK, but + * https://datatracker.ietf.org/doc/html/rfc7862#section-15.11.3 + * states "If the sa_offset is beyond the end of the file, then + * SEEK MUST return NFS4ERR_NXIO." + * + * Question is... which offset should a conforming NFSv4.2 + * SEEK_DATA return if there is no data block (i.e. sparse + * file which only consists of one hole) ? + */ +#define LINUX_NFSD_SEEK_NXIO_BUG_WORKAROUND 1 + +#ifdef LINUX_NFSD_SEEK_NXIO_BUG_WORKAROUND + if (data_seek_status == NFS4ERR_NXIO) { + DPRINTF(QARLVL, ("SEEK_DATA failed with NFS4ERR_NXIO\n")); + goto out; + } +#endif if (data_seek_status) { status = nfs_to_windows_error(data_seek_status, ERROR_INVALID_PARAMETER); diff --git a/tests/sparsefiles/testsparsefile1.ksh b/tests/sparsefiles/testsparsefile1.ksh index ea43f74..2860165 100644 --- a/tests/sparsefiles/testsparsefile1.ksh +++ b/tests/sparsefiles/testsparsefile1.ksh @@ -31,12 +31,69 @@ # +function test_sparse_holeonly +{ + set -o errexit + set -o nounset + #set -o xtrace -PATH='/bin:/usr/bin' + rm -f 'sparse_file_hole_only' + dd if='/dev/null' of='sparse_file_hole_only' bs=1 count=1 seek=$((65536*1024)) -builtin rm + ls -l 'sparse_file_hole_only' + /cygdrive/c/Windows/system32/fsutil sparse queryrange 'sparse_file_hole_only' + + integer fsutil_num_data_sections="$(/cygdrive/c/Windows/system32/fsutil sparse queryrange 'sparse_file_hole_only' | wc -l)" + + # + # test whether the file is OK + # + if (( fsutil_num_data_sections != 0 )) ; then + printf "# TEST failed, found %d data sections, expceted %d\n" \ + fsutil_num_data_sections \ + 0 + return 1 + fi + + printf "\n#\n# TEST %q OK, found %d data sections\n#\n" \ + "$0" \ + fsutil_num_data_sections + + return 0 +} + +function test_normal_file +{ + set -o errexit + set -o nounset + #set -o xtrace + + rm -f 'test_normal_file' + dd if='/dev/zero' of='test_normal_file' bs=1024 count=1024 + + ls -l 'test_normal_file' + /cygdrive/c/Windows/system32/fsutil sparse queryrange 'test_normal_file' -function test_sparsefile1 + integer fsutil_num_data_sections="$(/cygdrive/c/Windows/system32/fsutil sparse queryrange 'test_normal_file' | wc -l)" + + # + # test whether the file is OK + # + if (( fsutil_num_data_sections != 1 )) ; then + printf "# TEST failed, found %d data sections, expceted %d\n" \ + fsutil_num_data_sections \ + 0 + return 1 + fi + + printf "\n#\n# TEST %q OK, found %d data sections\n#\n" \ + "$0" \ + fsutil_num_data_sections + + return 0 +} + +function test_multihole_sparsefile1 { set -o errexit set -o nounset @@ -47,6 +104,7 @@ function test_sparsefile1 integer c.fsblocksize=$1 integer c.start_data_section=$2 integer c.end_data_section=$3 + typeset c.holeatend=$4 integer i compound -a c.filecontent @@ -61,7 +119,6 @@ function test_sparsefile1 ) done - # # generate sparse file # @@ -72,13 +129,28 @@ function test_sparsefile1 dd of='mysparsefile' bs=1 conv=notrunc seek=${c.filecontent[$i].pos} status=none <<<"${c.filecontent[$i].data}" done + # if we want a hole at the end, make a hole so the file itself is 8GB large + if ${c.holeatend} ; then + integer new_filesize=8*1024*1024*1024 + integer stat_filsize + + truncate -s ${new_filesize} 'mysparsefile' + + stat_filsize=$(stat --printf '%s\n' 'mysparsefile') + + if (( new_filesize != stat_filsize )) ; then + printf 'Filesize after extening via truncate -s %d, expected %d\n' \ + stat_filsize new_filesize + return 1 + fi + fi # # print results # printf '#\n# Results:\n#\n' - ls -l mysparsefile + ls -l 'mysparsefile' /cygdrive/c/Windows/system32/fsutil sparse queryrange 'mysparsefile' @@ -95,7 +167,8 @@ function test_sparsefile1 return 1 fi - printf "\n#\n# TEST OK, found %d data sections\n#\n" \ + printf "\n#\n# TEST %q OK, found %d data sections\n#\n" \ + "$0" \ fsutil_num_data_sections return 0 } @@ -105,14 +178,29 @@ function test_sparsefile1 # main # set -o errexit -test_sparsefile1 1024 0 4 -test_sparsefile1 1024 1 4 -test_sparsefile1 1024 0 32 -test_sparsefile1 1024 2 32 + +PATH='/bin:/usr/bin' + +builtin basename +builtin rm +builtin wc + +test_sparse_holeonly +test_normal_file + +test_multihole_sparsefile1 1024 0 4 false +test_multihole_sparsefile1 1024 1 4 false +test_multihole_sparsefile1 1024 0 32 false +test_multihole_sparsefile1 1024 2 32 false + +test_multihole_sparsefile1 1024 0 4 true +test_multihole_sparsefile1 1024 1 4 true # 512 does not work, as Win10 fsutil can only handle 64 data sections -# test_sparsefile1 1024 2 512 +# test_multihole_sparsefile1 1024 2 512 false + +printf '#\n# done\n#\n\n' -printf '%s: All tests OK\n' "$0" +printf '%s: All tests OK\n' "$(basename $0)" exit 0 # EOF. -- 2.45.1 From 68d0a7ab17dcafc5a9e657d3ea3cf4a23eeb50c5 Mon Sep 17 00:00:00 2001 From: Roland Mainz <roland.mainz@nrubsig.org> Date: Fri, 14 Feb 2025 13:41:43 +0100 Subject: [PATCH 2/3] RFE: Implement NFSv4.2 ALLOCATE+DEALLOCATE support Implement NFSv4.2 ALLOCATE+DEALLOCATE support for sparse file support. Reported-by: Lionel Cons <lionelcons1972@gmail.com> Reported-by: Cedric Blancher <cedric.blancher@gmail.com> Signed-off-by: Cedric Blancher <cedric.blancher@gmail.com> --- daemon/nfs41_ops.h | 38 +++++++++++ daemon/nfs41_xdr.c | 4 +- daemon/nfs41_xdr.h | 4 ++ daemon/nfs42_ops.c | 158 +++++++++++++++++++++++++++++++++++++++++++++ daemon/nfs42_xdr.c | 75 +++++++++++++++++++++ daemon/readwrite.c | 55 +++++++++++++++- daemon/recovery.c | 8 +++ 7 files changed, 338 insertions(+), 4 deletions(-) diff --git a/daemon/nfs41_ops.h b/daemon/nfs41_ops.h index f9bf445..bdb07a9 100644 --- a/daemon/nfs41_ops.h +++ b/daemon/nfs41_ops.h @@ -788,6 +788,28 @@ typedef struct __nfs42_read_plus_res { nfs42_read_plus_res_ok resok4; } nfs42_read_plus_res; +/* OP_ALLOCATE */ +typedef struct __nfs42_allocate_args { + stateid_arg *stateid; + uint64_t offset; + uint64_t length; +} nfs42_allocate_args; + +typedef struct __nfs42_allocate_res { + uint32_t status; +} nfs42_allocate_res; + +/* OP_DEALLOCATE */ +typedef struct __nfs42_deallocate_args { + stateid_arg *stateid; + uint64_t offset; + uint64_t length; +} nfs42_deallocate_args; + +typedef struct __nfs42_deallocate_res { + uint32_t status; +} nfs42_deallocate_res; + #if 1 /* OP_SEEK */ typedef struct __nfs42_seek_args { @@ -1203,6 +1225,22 @@ int nfs42_read_plus( OUT uint32_t *data_len_out, OUT bool_t *eof_out); +int nfs42_allocate( + IN nfs41_session *session, + IN nfs41_path_fh *file, + IN stateid_arg *stateid, + IN uint64_t offset, + IN uint64_t length, + OUT nfs41_file_info *cinfo); + +int nfs42_deallocate( + IN nfs41_session *session, + IN nfs41_path_fh *file, + IN stateid_arg *stateid, + IN uint64_t offset, + IN uint64_t length, + OUT nfs41_file_info *cinfo); + int nfs42_seek( IN nfs41_session *session, IN nfs41_path_fh *file, diff --git a/daemon/nfs41_xdr.c b/daemon/nfs41_xdr.c index afc4fbb..84c5f7f 100644 --- a/daemon/nfs41_xdr.c +++ b/daemon/nfs41_xdr.c @@ -3654,10 +3654,10 @@ static const op_table_entry g_op_table[] = { { encode_op_reclaim_complete, decode_op_reclaim_complete }, /* OP_RECLAIM_COMPLETE = 58 */ /* new operations for NFSv4.2 */ - { NULL, NULL }, /* OP_ALLOCATE = 59, */ + { encode_op_allocate, decode_op_allocate }, /* OP_ALLOCATE = 59, */ { NULL, NULL }, /* OP_COPY = 60, */ { NULL, NULL }, /* OP_COPY_NOTIFY = 61, */ - { NULL, NULL }, /* OP_DEALLOCATE = 62, */ + { encode_op_deallocate, decode_op_deallocate }, /* OP_DEALLOCATE = 62, */ { NULL, NULL }, /* OP_IO_ADVISE = 63, */ { NULL, NULL }, /* OP_LAYOUTERROR = 64, */ { NULL, NULL }, /* OP_LAYOUTSTATS = 65, */ diff --git a/daemon/nfs41_xdr.h b/daemon/nfs41_xdr.h index d6f217f..d89548c 100644 --- a/daemon/nfs41_xdr.h +++ b/daemon/nfs41_xdr.h @@ -34,6 +34,10 @@ void nfsacl41_free(nfsacl41 *acl); bool_t xdr_stateid4(XDR *xdr, stateid4 *si); /* NFSv4.2 ops */ +bool_t encode_op_allocate(XDR *xdr, nfs_argop4 *argop); +bool_t decode_op_allocate(XDR *xdr, nfs_resop4 *resop); +bool_t encode_op_deallocate(XDR *xdr, nfs_argop4 *argop); +bool_t decode_op_deallocate(XDR *xdr, nfs_resop4 *resop); bool_t encode_op_read_plus(XDR *xdr, nfs_argop4 *argop); bool_t decode_op_read_plus(XDR *xdr, nfs_resop4 *resop); bool_t encode_op_seek(XDR *xdr, nfs_argop4 *argop); diff --git a/daemon/nfs42_ops.c b/daemon/nfs42_ops.c index 39e946d..f416986 100644 --- a/daemon/nfs42_ops.c +++ b/daemon/nfs42_ops.c @@ -34,6 +34,164 @@ #include "daemon_debug.h" #include "util.h" +int nfs42_allocate( + IN nfs41_session *session, + IN nfs41_path_fh *file, + IN stateid_arg *stateid, + IN uint64_t offset, + IN uint64_t length, + OUT nfs41_file_info *cinfo) +{ + int status; + nfs41_compound compound; + nfs_argop4 argops[4]; + nfs_resop4 resops[4]; + nfs41_sequence_args sequence_args; + nfs41_sequence_res sequence_res; + nfs41_putfh_args putfh_args; + nfs41_putfh_res putfh_res; + nfs42_allocate_args allocate_args; + nfs42_allocate_res allocate_res; + nfs41_getattr_args getattr_args; + nfs41_getattr_res getattr_res = {0}; + bitmap4 attr_request; + nfs41_file_info info, *pinfo; + + nfs41_superblock_getattr_mask(file->fh.superblock, &attr_request); + + /* FIXME: What about DS in pNFS case ? */ + compound_init(&compound, session->client->root->nfsminorvers, + argops, resops, "allocate"); + + compound_add_op(&compound, OP_SEQUENCE, + &sequence_args, &sequence_res); + nfs41_session_sequence(&sequence_args, session, 0); + + compound_add_op(&compound, OP_PUTFH, &putfh_args, &putfh_res); + putfh_args.file = file; + putfh_args.in_recovery = 0; + + compound_add_op(&compound, OP_ALLOCATE, + &allocate_args, &allocate_res); + allocate_args.stateid = stateid; + allocate_args.offset = offset; + allocate_args.length = length; + + if (cinfo) { + pinfo = cinfo; + } + else { + (void)memset(&info, 0, sizeof(info)); + pinfo = &info; + } + + /* + * NFSv4.2 ALLOCATE is some kind of "write" operation and + * affects the number of physical bytes allocated, so we have + * to do a GETATTR after ALLOCATE to get updates for our cache + */ + compound_add_op(&compound, OP_GETATTR, &getattr_args, &getattr_res); + getattr_args.attr_request = &attr_request; + getattr_res.obj_attributes.attr_vals_len = NFS4_OPAQUE_LIMIT; + getattr_res.info = pinfo; + + status = compound_encode_send_decode(session, &compound, TRUE); + if (status) + goto out; + + if (compound_error(status = compound.res.status)) + goto out; + + /* update the attribute cache */ + bitmap4_cpy(&pinfo->attrmask, &getattr_res.obj_attributes.attrmask); + nfs41_attr_cache_update(session_name_cache(session), + file->fh.fileid, pinfo); + + nfs41_superblock_space_changed(file->fh.superblock); + +out: + return status; +} + +int nfs42_deallocate( + IN nfs41_session *session, + IN nfs41_path_fh *file, + IN stateid_arg *stateid, + IN uint64_t offset, + IN uint64_t length, + OUT nfs41_file_info *cinfo) +{ + int status; + nfs41_compound compound; + nfs_argop4 argops[4]; + nfs_resop4 resops[4]; + nfs41_sequence_args sequence_args; + nfs41_sequence_res sequence_res; + nfs41_putfh_args putfh_args; + nfs41_putfh_res putfh_res; + nfs42_deallocate_args deallocate_args; + nfs42_deallocate_res deallocate_res; + nfs41_getattr_args getattr_args; + nfs41_getattr_res getattr_res = {0}; + bitmap4 attr_request; + nfs41_file_info info, *pinfo; + + nfs41_superblock_getattr_mask(file->fh.superblock, &attr_request); + + /* FIXME: What about DS in pNFS case ? */ + compound_init(&compound, session->client->root->nfsminorvers, + argops, resops, "deallocate"); + + compound_add_op(&compound, OP_SEQUENCE, + &sequence_args, &sequence_res); + nfs41_session_sequence(&sequence_args, session, 0); + + compound_add_op(&compound, OP_PUTFH, &putfh_args, &putfh_res); + putfh_args.file = file; + putfh_args.in_recovery = 0; + + compound_add_op(&compound, OP_DEALLOCATE, + &deallocate_args, &deallocate_res); + deallocate_args.stateid = stateid; + deallocate_args.offset = offset; + deallocate_args.length = length; + + if (cinfo) { + pinfo = cinfo; + } + else { + (void)memset(&info, 0, sizeof(info)); + pinfo = &info; + } + + /* + * NFSv4.2 DEALLOCATE is some kind of "write" operation and + * affects the number of physical bytes allocated, so we have + * to do a GETATTR after DEALLOCATE to get updates for our cache + */ + compound_add_op(&compound, OP_GETATTR, &getattr_args, &getattr_res); + getattr_args.attr_request = &attr_request; + getattr_res.obj_attributes.attr_vals_len = NFS4_OPAQUE_LIMIT; + getattr_res.info = pinfo; + + status = compound_encode_send_decode(session, &compound, TRUE); + if (status) + goto out; + + if (compound_error(status = compound.res.status)) + goto out; + + /* update the attribute cache */ + bitmap4_cpy(&pinfo->attrmask, &getattr_res.obj_attributes.attrmask); + nfs41_attr_cache_update(session_name_cache(session), + file->fh.fileid, pinfo); + + nfs41_superblock_space_changed(file->fh.superblock); + +out: + return status; +} + int nfs42_read_plus( IN nfs41_session *session, IN nfs41_path_fh *file, diff --git a/daemon/nfs42_xdr.c b/daemon/nfs42_xdr.c index 9120461..210adb1 100644 --- a/daemon/nfs42_xdr.c +++ b/daemon/nfs42_xdr.c @@ -40,6 +40,81 @@ static __inline int unexpected_op(uint32_t op, uint32_t expected) return 1; } +/* + * OP_ALLOCATE + */ +bool_t encode_op_allocate( + XDR *xdr, + nfs_argop4 *argop) +{ + nfs42_allocate_args *args = (nfs42_allocate_args *)argop->arg; + + if (unexpected_op(argop->op, OP_ALLOCATE)) + return FALSE; + + if (!xdr_stateid4(xdr, &args->stateid->stateid)) + return FALSE; + + if (!xdr_u_hyper(xdr, &args->offset)) + return FALSE; + + return xdr_u_hyper(xdr, &args->length); +} + + +bool_t decode_op_allocate( + XDR *xdr, + nfs_resop4 *resop) +{ + nfs42_allocate_res *res = (nfs42_allocate_res *)resop->res; + + if (unexpected_op(resop->op, OP_ALLOCATE)) + return FALSE; + + if (!xdr_u_int32_t(xdr, &res->status)) + return FALSE; + + return TRUE; +} + +/* + * OP_DEALLOCATE + */ +bool_t encode_op_deallocate( + XDR *xdr, + nfs_argop4 *argop) +{ + nfs42_deallocate_args *args = (nfs42_deallocate_args *)argop->arg; + + if (unexpected_op(argop->op, OP_DEALLOCATE)) + return FALSE; + + if (!xdr_stateid4(xdr, &args->stateid->stateid)) + return FALSE; + + if (!xdr_u_hyper(xdr, &args->offset)) + return FALSE; + + return xdr_u_hyper(xdr, &args->length); +} + + +bool_t decode_op_deallocate( + XDR *xdr, + nfs_resop4 *resop) +{ + nfs42_deallocate_res *res = (nfs42_deallocate_res *)resop->res; + + if (unexpected_op(resop->op, OP_DEALLOCATE)) + return FALSE; + + if (!xdr_u_int32_t(xdr, &res->status)) + return FALSE; + + return TRUE; +} + + /* * OP_READ_PLUS */ diff --git a/daemon/readwrite.c b/daemon/readwrite.c index b285490..bce3e51 100644 --- a/daemon/readwrite.c +++ b/daemon/readwrite.c @@ -189,8 +189,9 @@ static int write_to_mds( IN nfs41_upcall *upcall, IN stateid_arg *stateid) { - nfs41_session *session = upcall->state_ref->session; - nfs41_path_fh *file = &upcall->state_ref->file; + nfs41_open_state *state = upcall->state_ref; + nfs41_session *session = state->session; + nfs41_path_fh *file = &state->file; readwrite_upcall_args *args = &upcall->args.rw; nfs41_write_verf verf; enum stable_how4 stable, committed; @@ -204,6 +205,56 @@ static int write_to_mds( (void)memset(&info, 0, sizeof(info)); + +#ifdef TEST_OP_ALLOCAE_OP_DEALLOCATE + /* + * Test code for OP_ALLOCATE and OP_DEALLOCATE, do not use except for + * testing! + */ + size_t data_i; + + /* Test whether the data block consists is a block of zero bytes */ + for (data_i = 0 ; data_i < args->len ; data_i++) { + if (((char *)args->buffer)[data_i] != '\0') + break; + } + + if (data_i == args->len) { + DPRINTF(0, ("write_to_mds(state->path.path='%s'): " + "Using DEALLOCATE+ALLOCATE for zero block\n", + state->path.path)); + + status = nfs42_deallocate(session, file, stateid, + args->offset, args->len, + &info); + if (status) { + DPRINTF(0, ("write_to_mds(state->path.path='%s'): " + "DEALLOCATE failed with status=0x%x\n", + state->path.path, + status)); + } + else { + status = nfs42_allocate(session, file, stateid, + args->offset, args->len, + &info); + if (status) { + DPRINTF(0, ("write_to_mds(state->path.path='%s'): " + "ALLOCATE failed with status=0x%x\n", + state->path.path, + status)); + } + } + + if (!status) { + /* Update ctime on success */ + args->ctime = info.change; + } + + len = args->len; + goto out; + } +#endif /* TEST_OP_ALLOCAE_OP_DEALLOCATE */ + retry_write: p = args->buffer; to_send = args->len; diff --git a/daemon/recovery.c b/daemon/recovery.c index 42ca27b..66bf25e 100644 --- a/daemon/recovery.c +++ b/daemon/recovery.c @@ -809,6 +809,14 @@ bool_t nfs41_recover_stateid( } else if (argop->op == OP_READ_PLUS) { nfs42_read_plus_args *read_plus = (nfs42_read_plus_args *)argop->arg; stateid = read_plus->stateid; + } else if (argop->op == OP_ALLOCATE) { + nfs42_allocate_args *allocate = + (nfs42_allocate_args *)argop->arg; + stateid = allocate->stateid; + } else if (argop->op == OP_DEALLOCATE) { + nfs42_deallocate_args *deallocate = + (nfs42_deallocate_args *)argop->arg; + stateid = deallocate->stateid; } else if (argop->op == OP_SEEK) { nfs42_seek_args *seek = (nfs42_seek_args *)argop->arg; stateid = seek->stateid; -- 2.45.1 From 0c8b3f29d53061b6db4704b22a098d89041d35e5 Mon Sep 17 00:00:00 2001 From: Roland Mainz <roland.mainz@nrubsig.org> Date: Fri, 14 Feb 2025 16:59:13 +0100 Subject: [PATCH 3/3] daemon,include,sys,tests: Implement |FSCTL_SET_ZERO_DATA| via NFSv4.2 DEALLOCATE Implement |FSCTL_SET_ZERO_DATA| via NFSv4.2 DEALLOCATE, which enables utilities like Cygwin /usr/bin/fallocate to punch holes into files. Reported-by: Lionel Cons <lionelcons1972@gmail.com> Reported-by: Cedric Blancher <cedric.blancher@gmail.com> Signed-off-by: Cedric Blancher <cedric.blancher@gmail.com> --- daemon/daemon_debug.c | 1 + daemon/fsctl.c | 117 +++++++++++++++++++++ daemon/upcall.c | 2 + daemon/upcall.h | 5 + include/from_kernel.h | 12 +++ include/nfs41_driver.h | 1 + sys/nfs41sys_debug.c | 1 + sys/nfs41sys_driver.h | 11 ++ sys/nfs41sys_fsctl.c | 141 ++++++++++++++++++++++++++ sys/nfs41sys_updowncall.c | 7 ++ tests/sparsefiles/testsparsefile1.ksh | 40 ++++++++ 11 files changed, 338 insertions(+) diff --git a/daemon/daemon_debug.c b/daemon/daemon_debug.c index ab7f567..4f3a9f2 100644 --- a/daemon/daemon_debug.c +++ b/daemon/daemon_debug.c @@ -467,6 +467,7 @@ const char* opcode2string(nfs41_opcodes opcode) NFSOPCODE_TO_STRLITERAL(NFS41_SYSOP_ACL_QUERY) NFSOPCODE_TO_STRLITERAL(NFS41_SYSOP_ACL_SET) NFSOPCODE_TO_STRLITERAL(NFS41_SYSOP_FSCTL_QUERYALLOCATEDRANGES) + NFSOPCODE_TO_STRLITERAL(NFS41_SYSOP_FSCTL_SET_ZERO_DATA) NFSOPCODE_TO_STRLITERAL(NFS41_SYSOP_INVALID_OPCODE1) default: break; } diff --git a/daemon/fsctl.c b/daemon/fsctl.c index 421909d..dfff5a4 100644 --- a/daemon/fsctl.c +++ b/daemon/fsctl.c @@ -28,6 +28,7 @@ #include "util.h" #define QARLVL 2 /* dprintf level for "query allocated ranges" logging */ +#define SZDLVL 2 /* dprintf level for "set zero data" logging */ static int parse_queryallocatedranges(unsigned char *buffer, uint32_t length, nfs41_upcall *upcall) @@ -254,3 +255,119 @@ const nfs41_upcall_op nfs41_op_queryallocatedranges = { .marshall = marshall_queryallocatedranges, .arg_size = sizeof(queryallocatedranges_upcall_args) }; + +static int parse_setzerodata(unsigned char *buffer, + uint32_t length, nfs41_upcall *upcall) +{ + int status; + setzerodata_upcall_args *args = &upcall->args.setzerodata; + + status = safe_read(&buffer, &length, &args->setzerodata, + sizeof(args->setzerodata)); + if (status) goto out; + + DPRINTF(SZDLVL, ("parse_setzerodata: " + "parsing '%s' setzerodata=(FileOffset=%lld BeyondFinalZero=%lld)\n", + opcode2string(upcall->opcode), + (long long)args->setzerodata.FileOffset.QuadPart, + (long long)args->setzerodata.BeyondFinalZero.QuadPart)); +out: + return status; +} + + +static +int handle_setzerodata(void *daemon_context, + nfs41_upcall *upcall) +{ + int status = ERROR_INVALID_PARAMETER; + setzerodata_upcall_args *args = &upcall->args.setzerodata; + nfs41_open_state *state = upcall->state_ref; + nfs41_session *session = state->session; + nfs41_path_fh *file = &state->file; + nfs41_file_info info; + int64_t offset_start; /* signed! */ + int64_t offset_end; /* signed! */ + int64_t len; /* signed! */ + stateid_arg stateid; + + (void)memset(&info, 0, sizeof(info)); + + offset_start = args->setzerodata.FileOffset.QuadPart; + offset_end = args->setzerodata.BeyondFinalZero.QuadPart; + len = offset_end - offset_start; + + DPRINTF(SZDLVL, + ("--> handle_setzerodata(" + "state->path.path='%s', " + "offset_start=%lld, " + "offset_end=%lld, " + "len=%lld)\n", + state->path.path, + offset_start, + offset_end, + len)); + + /* NFS DEALLOCATE requires NFSv4.2 */ + if (state->session->client->root->nfsminorvers < 2) { + status = ERROR_NOT_SUPPORTED; + goto out; + } + + if (len < 0) { + status = ERROR_INVALID_PARAMETER; + DPRINTF(SZDLVL, ("handle_setzerodata: invalid len\b")); + goto out; + } + + if (len == 0) { + status = NO_ERROR; + DPRINTF(SZDLVL, ("handle_setzerodata: len == 0, NOP\n")); + goto out; + } + + nfs41_open_stateid_arg(state, &stateid); + + status = nfs42_deallocate(session, file, &stateid, + offset_start, len, &info); + if (status) { + DPRINTF(SZDLVL, ("handle_setzerodata(state->path.path='%s'): " + "DEALLOCATE failed with status=0x%x\n", + state->path.path, + status)); + goto out; + } + + /* Update ctime on success */ + EASSERT((info.attrmask.count > 0) && + (info.attrmask.arr[0] & FATTR4_WORD0_CHANGE)); + args->ctime = info.change; + + DPRINTF(SZDLVL, + ("handle_setzerodata(state->path.path='%s'): args->ctime=%llu\n", + state->path.path, + args->ctime)); + +out: + DPRINTF(SZDLVL, + ("<-- handle_setzerodata(), status=0x%lx\n", + status)); + + return status; +} + +static int marshall_setzerodata(unsigned char *buffer, + uint32_t *length, nfs41_upcall *upcall) +{ + setzerodata_upcall_args *args = &upcall->args.setzerodata; + int status; + status = safe_write(&buffer, length, &args->ctime, sizeof(args->ctime)); + return status; +} + +const nfs41_upcall_op nfs41_op_setzerodata = { + .parse = parse_setzerodata, + .handle = handle_setzerodata, + .marshall = marshall_setzerodata, + .arg_size = sizeof(setzerodata_upcall_args) +}; diff --git a/daemon/upcall.c b/daemon/upcall.c index 3f741e6..8643fb5 100644 --- a/daemon/upcall.c +++ b/daemon/upcall.c @@ -50,6 +50,7 @@ extern const nfs41_upcall_op nfs41_op_volume; extern const nfs41_upcall_op nfs41_op_getacl; extern const nfs41_upcall_op nfs41_op_setacl; extern const nfs41_upcall_op nfs41_op_queryallocatedranges; +extern const nfs41_upcall_op nfs41_op_setzerodata; /* |_nfs41_opcodes| and |g_upcall_op_table| must be in sync! */ static const nfs41_upcall_op *g_upcall_op_table[] = { @@ -74,6 +75,7 @@ static const nfs41_upcall_op *g_upcall_op_table[] = { &nfs41_op_getacl, &nfs41_op_setacl, &nfs41_op_queryallocatedranges, + &nfs41_op_setzerodata, NULL, NULL }; diff --git a/daemon/upcall.h b/daemon/upcall.h index 290aabd..94f830c 100644 --- a/daemon/upcall.h +++ b/daemon/upcall.h @@ -197,6 +197,10 @@ typedef struct __queryallocatedranges_upcall_args { ULONG returned_size; } queryallocatedranges_upcall_args; +typedef struct __setzerodata_upcall_args { + FILE_ZERO_DATA_INFORMATION setzerodata; + ULONGLONG ctime; +} setzerodata_upcall_args; typedef union __upcall_args { mount_upcall_args mount; @@ -215,6 +219,7 @@ typedef union __upcall_args { getacl_upcall_args getacl; setacl_upcall_args setacl; queryallocatedranges_upcall_args queryallocatedranges; + setzerodata_upcall_args setzerodata; } upcall_args; typedef enum _nfs41_opcodes nfs41_opcodes; diff --git a/include/from_kernel.h b/include/from_kernel.h index 8b9ddc0..7a9c47a 100644 --- a/include/from_kernel.h +++ b/include/from_kernel.h @@ -452,4 +452,16 @@ typedef struct _FILE_ALLOCATED_RANGE_BUFFER { } FILE_ALLOCATED_RANGE_BUFFER, *PFILE_ALLOCATED_RANGE_BUFFER; #endif /* _WINIOCTL_ */ +/* + * |FILE_ZERO_DATA_INFORMATION| - test for + * /usr/i686-w64-mingw32/sys-root/mingw/include/winioctl.h header + * to avoid type redefinition warnings + */ +#ifndef _WINIOCTL_ +typedef struct _FILE_ZERO_DATA_INFORMATION { + LARGE_INTEGER FileOffset; /* absolute offset */ + LARGE_INTEGER BeyondFinalZero; /* absolute offset */ +} FILE_ZERO_DATA_INFORMATION, *PFILE_ZERO_DATA_INFORMATION; +#endif /* !_WINIOCTL_ */ + #endif diff --git a/include/nfs41_driver.h b/include/nfs41_driver.h index 403bf04..37b3141 100644 --- a/include/nfs41_driver.h +++ b/include/nfs41_driver.h @@ -83,6 +83,7 @@ typedef enum _nfs41_opcodes { NFS41_SYSOP_ACL_QUERY, NFS41_SYSOP_ACL_SET, NFS41_SYSOP_FSCTL_QUERYALLOCATEDRANGES, + NFS41_SYSOP_FSCTL_SET_ZERO_DATA, NFS41_SYSOP_SHUTDOWN, NFS41_SYSOP_INVALID_OPCODE1 } nfs41_opcodes; diff --git a/sys/nfs41sys_debug.c b/sys/nfs41sys_debug.c index dccd933..dd35f90 100644 --- a/sys/nfs41sys_debug.c +++ b/sys/nfs41sys_debug.c @@ -678,6 +678,7 @@ const char *opcode2string(int opcode) case NFS41_SYSOP_ACL_QUERY: return "NFS41_SYSOP_ACL_QUERY"; case NFS41_SYSOP_ACL_SET: return "NFS41_SYSOP_ACL_SET"; case NFS41_SYSOP_FSCTL_QUERYALLOCATEDRANGES: return "NFS41_SYSOP_FSCTL_QUERYALLOCATEDRANGES"; + case NFS41_SYSOP_FSCTL_SET_ZERO_DATA: return "NFS41_SYSOP_FSCTL_SET_ZERO_DATA"; default: return "UNKNOWN"; } } diff --git a/sys/nfs41sys_driver.h b/sys/nfs41sys_driver.h index 7fccf69..f14d33d 100644 --- a/sys/nfs41sys_driver.h +++ b/sys/nfs41sys_driver.h @@ -276,6 +276,9 @@ typedef struct _updowncall_entry { PVOID Buffer; LONGLONG returned_size; } QueryAllocatedRanges; + struct { + FILE_ZERO_DATA_INFORMATION setzerodata; + } SetZeroData; } u; } nfs41_updowncall_entry; @@ -650,6 +653,14 @@ NTSTATUS marshal_nfs41_queryallocatedranges( NTSTATUS unmarshal_nfs41_queryallocatedranges( nfs41_updowncall_entry *cur, unsigned char **buf); +NTSTATUS marshal_nfs41_setzerodata( + nfs41_updowncall_entry *entry, + unsigned char *buf, + ULONG buf_len, + ULONG *len); +NTSTATUS unmarshal_nfs41_setzerodata( + nfs41_updowncall_entry *cur, + unsigned char **buf); /* nfs41sys_ioctl.c */ NTSTATUS nfs41_IoCtl( diff --git a/sys/nfs41sys_fsctl.c b/sys/nfs41sys_fsctl.c index 1799fe9..acacf7e 100644 --- a/sys/nfs41sys_fsctl.c +++ b/sys/nfs41sys_fsctl.c @@ -368,6 +368,144 @@ out: return status; } +static +NTSTATUS nfs41_SetZeroData( + IN OUT PRX_CONTEXT RxContext) +{ + NTSTATUS status = STATUS_INVALID_DEVICE_REQUEST; + nfs41_updowncall_entry *entry = NULL; + __notnull PMRX_SRV_OPEN SrvOpen = RxContext->pRelevantSrvOpen; + __notnull PNFS41_V_NET_ROOT_EXTENSION pVNetRootContext = + NFS41GetVNetRootExtension(SrvOpen->pVNetRoot); + __notnull PNFS41_NETROOT_EXTENSION pNetRootContext = + NFS41GetNetRootExtension(SrvOpen->pVNetRoot->pNetRoot); + __notnull XXCTL_LOWIO_COMPONENT *FsCtl = + &RxContext->LowIoContext.ParamsFor.FsCtl; + __notnull PFILE_ZERO_DATA_INFORMATION setzerodatabuffer = + (PFILE_ZERO_DATA_INFORMATION)FsCtl->pInputBuffer; + __notnull PNFS41_FOBX nfs41_fobx = NFS41GetFobxExtension(RxContext->pFobx); + + DbgEn(); + + RxContext->IoStatusBlock.Information = 0; + + if (FsCtl->InputBufferLength < + sizeof(FILE_ZERO_DATA_INFORMATION)) { + DbgP("nfs41_SetZeroData: " + "buffer to small\n"); + status = STATUS_BUFFER_TOO_SMALL; + goto out; + } + + DbgP("nfs41_SetZeroData: " + "setzerodatabuffer=(FileOffset=%lld,BeyondFinalZero=%lld)\n", + (long long)setzerodatabuffer->FileOffset.QuadPart, + (long long)setzerodatabuffer->BeyondFinalZero.QuadPart); + + /* + * Disable caching because NFSv4.2 DEALLOCATE is basically a + * "write" operation. AFAIK we should flush the cache and wait + * for the kernel lazy writer (which |RxChangeBufferingState()| + * AFAIK does) before doing the DEALLOCATE, to avoid that we + * have outstanding writes in the kernel cache at the same + * location where the DEALLOCATE should do it's work + */ + ULONG flag = DISABLE_CACHING; + DbgP("nfs41_SetZeroData: disableing caching for file '%wZ'\n", + SrvOpen->pAlreadyPrefixedName); + RxChangeBufferingState((PSRV_OPEN)SrvOpen, ULongToPtr(flag), 1); + + status = nfs41_UpcallCreate(NFS41_SYSOP_FSCTL_SET_ZERO_DATA, + &nfs41_fobx->sec_ctx, + pVNetRootContext->session, + nfs41_fobx->nfs41_open_state, + pNetRootContext->nfs41d_version, + SrvOpen->pAlreadyPrefixedName, + &entry); + + if (status) + goto out; + + entry->u.SetZeroData.setzerodata = *setzerodatabuffer; + + status = nfs41_UpcallWaitForReply(entry, pVNetRootContext->timeout); + if (status) { + /* Timeout - |nfs41_downcall()| will free |entry|+contents */ + goto out; + } + + if (entry->psec_ctx == &entry->sec_ctx) { + SeDeleteClientSecurity(entry->psec_ctx); + } + entry->psec_ctx = NULL; + + if (!entry->status) { + DbgP("nfs41_SetZeroData: SUCCESS\n"); + RxContext->CurrentIrp->IoStatus.Status = STATUS_SUCCESS; + RxContext->IoStatusBlock.Information = 0; + } + else { + DbgP("nfs41_SetZeroData: " + "FAILURE, entry->status=0x%lx\n", entry->status); + status = map_setfile_error(entry->status); + RxContext->CurrentIrp->IoStatus.Status = status; + RxContext->IoStatusBlock.Information = 0; + } + + if (entry) { + nfs41_UpcallDestroy(entry); + } + +out: + DbgEx(); + return status; +} + +NTSTATUS marshal_nfs41_setzerodata( + nfs41_updowncall_entry *entry, + unsigned char *buf, + ULONG buf_len, + ULONG *len) +{ + NTSTATUS status = STATUS_SUCCESS; + ULONG header_len = 0; + unsigned char *tmp = buf; + + status = marshal_nfs41_header(entry, tmp, buf_len, len); + if (status) goto out; + else tmp += *len; + + header_len = *len + sizeof(FILE_ZERO_DATA_INFORMATION); + if (header_len > buf_len) { + status = STATUS_INSUFFICIENT_RESOURCES; + goto out; + } + + RtlCopyMemory(tmp, &entry->u.SetZeroData.setzerodata, + sizeof(entry->u.SetZeroData.setzerodata)); + tmp += sizeof(entry->u.SetZeroData.setzerodata); + + *len = header_len; + + DbgP("marshal_nfs41_setzerodata: name='%wZ'\n", + entry->filename); +out: + return status; +} + +NTSTATUS unmarshal_nfs41_setzerodata( + nfs41_updowncall_entry *cur, + unsigned char **buf) +{ + NTSTATUS status = STATUS_SUCCESS; + + RtlCopyMemory(&cur->ChangeTime, *buf, sizeof(ULONGLONG)); + DbgP("unmarshal_nfs41_setzerodata: returned ChangeTime %llu\n", + cur->ChangeTime); + + return status; +} + NTSTATUS nfs41_FsCtl( IN OUT PRX_CONTEXT RxContext) { @@ -392,6 +530,9 @@ NTSTATUS nfs41_FsCtl( case FSCTL_SET_SPARSE: status = nfs41_SetSparse(RxContext); break; + case FSCTL_SET_ZERO_DATA: + status = nfs41_SetZeroData(RxContext); + break; default: break; } diff --git a/sys/nfs41sys_updowncall.c b/sys/nfs41sys_updowncall.c index 5d1d259..e9e6eac 100644 --- a/sys/nfs41sys_updowncall.c +++ b/sys/nfs41sys_updowncall.c @@ -306,6 +306,10 @@ NTSTATUS handle_upcall( status = marshal_nfs41_queryallocatedranges(entry, pbOut, cbOut, len); break; + case NFS41_SYSOP_FSCTL_SET_ZERO_DATA: + status = marshal_nfs41_setzerodata(entry, + pbOut, cbOut, len); + break; default: status = STATUS_INVALID_PARAMETER; print_error("Unknown nfs41 ops %d\n", entry->opcode); @@ -667,6 +671,9 @@ NTSTATUS nfs41_downcall( case NFS41_SYSOP_FSCTL_QUERYALLOCATEDRANGES: unmarshal_nfs41_queryallocatedranges(cur, &buf); break; + case NFS41_SYSOP_FSCTL_SET_ZERO_DATA: + unmarshal_nfs41_setzerodata(cur, &buf); + break; } } ExReleaseFastMutex(&cur->lock); diff --git a/tests/sparsefiles/testsparsefile1.ksh b/tests/sparsefiles/testsparsefile1.ksh index 2860165..fa0eef5 100644 --- a/tests/sparsefiles/testsparsefile1.ksh +++ b/tests/sparsefiles/testsparsefile1.ksh @@ -173,6 +173,44 @@ function test_multihole_sparsefile1 return 0 } +function test_sparse_punchhole1 +{ + set -o errexit + set -o nounset + #set -o xtrace + + rm -f 'sparse_file_punchhole' + dd if='/dev/zero' of='sparse_file_punchhole' count=8 bs=$((1024*1024)) status=none + chattr -V +S 'sparse_file_punchhole' + + printf '# expected: one data section before fallocate\n' + /cygdrive/c/Windows/system32/fsutil sparse queryrange 'sparse_file_punchhole' + + fallocate -n -p -o $((0x16000)) -l $((0x8000)) 'sparse_file_punchhole' + + printf '# expected: two data section after fallocate\n' + /cygdrive/c/Windows/system32/fsutil sparse queryrange 'sparse_file_punchhole' + + integer fsutil_num_data_sections="$(/cygdrive/c/Windows/system32/fsutil sparse queryrange 'sparse_file_punchhole' | wc -l)" + + # + # test whether the file is OK + # + if (( fsutil_num_data_sections != 2 )) ; then + printf "# TEST %q failed, found %d data sections, expceted %d\n" \ + "$0" \ + fsutil_num_data_sections \ + 2 + return 1 + fi + + printf "\n#\n# TEST %q OK, found %d data sections\n#\n" \ + "$0" \ + fsutil_num_data_sections + + return 0 +} + # # main @@ -199,6 +237,8 @@ test_multihole_sparsefile1 1024 1 4 true # 512 does not work, as Win10 fsutil can only handle 64 data sections # test_multihole_sparsefile1 1024 2 512 false +test_sparse_punchhole1 + printf '#\n# done\n#\n\n' printf '%s: All tests OK\n' "$(basename $0)" -- 2.45.1

Title:

Privacy: Public Private

How long should your post be retained?

15 mins an hour a day a week a month forever

All content is user-submitted.
The administrators of this site (kpaste.net) are not responsible for their content.
Abuse reports should be emailed to us at