Objective: To use the Short Form 36 (SF-36) to determine the extent to which health-related quality of life (HRQOL) is decreased among primary caregivers of patients with cerebrovascular accident (CVA) or diabetes mellitus (DM) compared to normal Taiwanese population and to identify the determinants of this decrease. Methods: Data from a cross-sectional survey of 187 primary caregivers who had responsibility for inpatients with a medically verified diagnosis of CVA or DM were compared to those of randomly selected residents. Data were collected by face-to-face interviews with trained interviewers. Results: The age- and gender-adjusted mean differences of caregivers on primarily mental scales of SF-36 were significantly negative compared to general population, as much as a 3–12 points reduction on this 100-point scale. While the age- and gender-adjusted mean differences on physical functioning and bodily pain scales were significantly positive, as much as a 3–6 points increase. Effects of caregiving on the perceived social life of the caregiver and disability of inpatients in eating and getting in/out of bed were associated with the SF-36 Physical Component Scale (PCS) score while female gender, type of caregiver, care conflicts, degree of care demand of daily living, and effects of caregivering on perceived social life of caregivers were negatively associated with the SF-36 Mental Component Scale (MCS) score. Conclusions: Primary caregivers of CVA or DM hospitalized elderly have poorer mental but better physical well-being than the population norm. Both caregiver and inpatient factors contribute to caregivers'' HRQOL.