nf_conntrack entry lost after using ipq_set_verdict() with NF_ACCEPT

I am going to conntrack and NAT for a udp traffic with destination port 4.
in the conntrack helper() i did so:

if (ct->status & IPS_NAT_MASK){
    printk("going to ALG part after NAT\n");
    ret = NF_QUEUE;
    }

The first query (with dport 4) packet after MASQUERADE goes into the application layer gateway with NF_QUEUE into userspace, where the payload can be further changed.
Then i use ipq_set_verdict() with verdict NF_ACCEPT to forward it. But when the response in the other direction comes, it cannot be recognized as the reply of the query.
I think the entry of nf_conntrack of the first query packet has been lost, because when i just use MASQUERADE without userspace processing, the response will be marked correct as IP_CT_IS_REPLY.

I have printed out the conntrack tuple when the helper()is called, for the Query,
ctinfo = IP_CT_NEW;
the original tuple is:
10.21.22.21:4 -> 10.23.24.24:4 l3num:2 protonum:17
the reply tuple:
10.23.24.24:4 -> 10.22.23.22:4 l3num:2 protonum:17 10.21.22.21 is the ip address of querying node, 10.22.23.22 is the NAT and 10.23.24.24 is the responding node.

As for the Response:
ctinfo = IP_CT_NEW(which should be IP_CT_IS_REPLY) original tuple:
10.23.24.24:4 -> 10.22.23.22:4 l3num:2 protonum:17 reply tuple:
10.22.23.22:4 -> 10.23.24.24:4 l3num:2 protonum:17

Is there someone could help me figure this out??

best regards!!