We propose a class of longitudinal data models with random effects that generalizes currently used models in two important aspects. First, the random effects model is a flexible mixture of multivariate normals, accomodating population heterogeneity, outliers and non-linearity in regression on subject-specific covariates. Second, the model includes a hierarchical extension to allow for meta-analysis over related studies. The random effects distributions are decomposed into one part that is common across all related studies (common measure), and one part that is specific to each study and captures the variability intrinsic within patients from the same study. Both, the common measure and the study-specific measures, are parameterized as mixture of normal models. We introduce a reversible jump algorithm for posterior simulation to allow random number of terms in the mixtures. Our sampler takes advantage of the small number of entertained models when elaborating on proposal distributions. The motivating application is the analysis of two studies carried out by the Cancer and Leukemia Group B (CALGB). In both studies, we record for each patient white blood cell count (WBC) over time to characterize the toxic effects of treatment. The WBCs are modeled through a nonlinear hierarchical model that gathers the information from both studies.

Keywords: Markov chain Monte Carlo, mixture model, model averaging, model selection, pharmacodynamic models, reversible jump.

The current version of the manuscript is available in postscript format.