The nonlinear magnetic susceptibility $\chi$($H_{\textrm{ac}}$) and critical current $J_{\textrm{c}}$ in thin YBa$_2$Cu$_3$O$_{7-\delta}$ films are measured using planar spiral open-faced pick-up coils. The real and imaginary parts of a generalized susceptibility $\tilde{\chi}(a, h)$ are calculated within the framework of the critical state model for an arbitrary ratio $a = r/R$ of the radius of a single-turn pick-up coil $r$ to the superconducting thin-film disc radius $R$. Well known results of the Clem–Sanchez model in the limit $a \to \infty$ are reproduced for nonlinear complex magnetic susceptibility $\chi$($h$), where $h$($J_{\textrm{c}}$($T$), $H_{\textrm{ac}}$) is reduced dimensionless field amplitude. Relative systematic error of measured $J_{\textrm{c}}$ is shown to be negligible (< 2%) in a standard experimental geometry (with a sample within the pick-up coil of magnetometer) for $a > 2$, but it is of about 25% at $a \to 1$. On the other hand, for the open-faced coils with $a < 1$, the nature of amplitude dependences of $\tilde{\chi}$($a$, $h$) is changed qualitatively, turning to have threshold on $h$. That is why, at the fixed field amplitude $H_{\textrm{ac}}$, not unique point of the $J_{\textrm{c}}$($T$) dependence (at the maximum of $\chi^{''}$($T$, $H_{\textrm{ac}}$)) but two points (at both the maximum and the threshold of $\tilde{\chi}$$^{''}$($T$, $H_{\textrm{ac}}$)–$T$ dependence) can be obtained. This also simplifies the normalization procedure of the real and imaginary parts of generalized magnetic susceptibility.