Abstract: Adapter-based tuning is a solid training method for adapting pre-trained language models to various natural language understanding tasks. Most existing studies are focused on adding learnable parameters between Transformer layers, which increases the layer number in the model and suffers from gradient instability and lower convergence speed. This paper proposes a lightweight adapter in feed-forward network for natural language understanding, named F-Adapter, which introduces a small number of hidden units inside each feed-forward network. Essentially, it expands the weight matrices and bias. Furthermore, this paper demonstrates that the impact of the trainable parameters in the weight matrix and bias (matrix) on model performance is independent of their positions within the matrix. The experiment results show that F-Adapter, with only about 3% of PLM parameters to be trained, outperforms all baselines on average across 11 datasets for 7 downstream tasks.Compared with baselines such as P-tuning v2 and Adapter, F-Adapter is more stable and converges faster.