A traditional idea of the Method of Fundamental Solutions is to use some external source points where the fundamental solution should be shifted to. However, the proper definition of the locations of the sources can hardly be performed in an automated way. To circumvent this difficulty, in this paper, the source points defined along the boundary, and the collocation points are shifted to the interior of the domain together with a proper modification of the boundary conditions. Thus, the problem of singularity is avoided. The modified boundary conditions are defined on the basis of the tools of the classical finite difference methods. Several schemes are presented. The schemes can be embedded in a multi-level context in a natural way. The proposed method avoids the computational difficulties due to ill-conditioned matrices and also reduces the computational complexity of the Method of Fundamental Solutions.