Pneumonia is one of the major diseases that cause a lot of deaths all over the world. Determining pneumonia from chest X-ray (CXR) images is an extremely difficult and important image processing problem. The discrimination of whether pneumonia is of bacterium or virus origin has also become more important during the pandemic. Automatic determination of the presence and origin of pneumonia is crucial for speeding up the treatment process and increasing the patient’s survival rate. In this study, a convolutional neural network (CNN) framework is proposed for detection of pneumonia from CXR images. Two different binary CNNs and a triple CNN are used for determining: (i) normal or pneumonia, (ii) pneumonia of bacterium or virus origin, and (iii) normal or bacterial pneumonia or viral pneumonia. In this approach, CNNs are trained with Walsh functions to extract the features from CXR images, and minimum distance classifier instead of a fully connected neural network is employed for classification purpose. Training with Walsh functions maintains the within-class scattering to be low, and between-class scattering to be high. Preferring the minimum distance classifier reduces the number of nodes used and also allows the network to be controlled with fewer hyperparameters. These approaches bring some advantages to the system designed for the classification process: (i) easy determination of hyperparameters, (ii) achieving higher classification performance, and (iii) use of fewer neurons. The proposed smallsize CNN model was applied to CXR images from 1- to 5-year-old children provided by the Guangzhou Women’s and Children’s Medical Center (GWCMC). Three experiments have been conducted to improve the classification performance: (i) the effect of different sizes of input images on the performance of the network was investigated, (ii) training set was augmented by randomly flipping left to right or down to up, by adding Gaussian noise to the images, by creating negative images randomly, and by changing image brightness randomly (iii) instead of RGB CXR images, gray component of the original image and four 2D wavelet images were given as input to the network. In these experiments, no major changes were observed in the classification results obtained by using the proposed CNNs. The proposed method has achieved 100% accuracy for normal or pneumonia, 92% for pneumonia of bacterium or virus origin, and 90% for normal or bacterial pneumonia or viral pneumonia. It is observed that higher classification performances were obtained with approximately five times less parameters compared to the networks that gave the best results in the literature. Thus, the applied CNN model is promising in medicine and can help experts make quick and accurate decisions.