Tea leaf blight (TLB) is a common tea disease seriously affecting the quality and yield of tea. An accurate estimation of TLB severity can be used to guide tea farmers to reasonably spray pesticides. This study proposes an estimation method for TLB severity in natural scene images and consists of four main steps: segmentation of the diseased leaves, area fitting of the diseased leaves, segmentation of the disease spots, and estimation of disease severity. Target leaves with TLB in the tea images are segmented by combining the U-Net network and fully connected conditional random field to reduce the influence of complex background. An ellipse restoration method is proposed to generate an elliptic mask to fit the full size of the occluded or damaged TLB leaves. The disease spot regions are segmented from the TLB leaves by a support vector machine classifier to calculate the Initial Disease Severity (IDS) index. The IDS index, color features, and texture features of the TLB leaves are inputted into the metric learning model to finally estimated disease severity. Experimental results show that the proposed method has higher estimation accuracy and stronger robustness against occluded and damaged TLB leaves compared with conventional convolution neural network methods and classical machine learning techniques.