This paper bridges the gap in the literature between neural networks and probabilistic graphical models. Invertible neural networks are incorporated in factor graphs and inference in this model is described by linearization of the network. Consequently, hybrid probabilistic inference in the model is realized through message passing with local constraints on the Bethe free energy. We provide the local Bethe free energy for the invertible neural network node, which allows for evaluation of the performance of the entire probabilistic model. Experimental results show effective hybrid inference in a neural network-based probabilistic model for a binary classification task, paving the way towards a novel class of machine learning models.