Re-sequencing permits the mining of genome-wide variations on a large scale and provides excellent resources for the research community. To accelerate the development and application of molecular markers and identify the QTLs affecting the flowering time-related trait in pepper, a total of 1,038 pairs of InDel and 674 SSR primers from different sources were used for genetic mapping using the F2 population (n = 154) derived from a cross between BA3 (C. annuum) and YNXML (C. frutescens). Of these, a total of 224 simple PCR-based markers, including 129 InDels and 95 SSRs, were validated and integrated into a map, which was designated as the BY map. The BY map consisted of 13 linkage groups (LGs) and spanned a total genetic distance of 1,249.77 cM with an average marker distance of 5.60 cM. Comparative analysis of the genetic and physical map based on the anchored markers showed that the BY map covered nearly the whole pepper genome. Based on the BY map, one major and five minor QTLs affecting the number of leaves on the primary axis (Nle) were detected on chromosomes P2, P7, P10 and P11 in 2012. The major QTL on P2 was confirmed based on another subset of the same F2 population (n = 147) in 2014 with selective genotyping of markers from the BY map. With the accomplishment of pepper whole genome sequencing and annotations (release 2.0), 153 candidate genes were predicted to embed in the Nle2.2 region, of which 12 important flowering related genes were obtained. The InDel/SSR-based interspecific genetic map, QTLs and candidate genes obtained by the present study will be useful for the downstream isolation of flowering time-related gene and other genetic applications for pepper.