Interlineage communication within a cancer microenvironment can augment cancer cell behaviour and impact response to therapy. Patient-derived cancer organoids provide an opportunity to explore cancer cell biology, however it is a major challenge to generate a complex cancer microenvironment in vitro. Here, we established a stromal tumoroid culture system modeling pancreatic ductal adenocarcinoma (PDAC) that reconstitutes multilineage interactions between cancer, endothelial, and fibroblast cells and recapitulates several aspects of primary tumors. Whole-mount immunohistochemistry on cleared tumoroids reveals organized vessel, desmoplastic fibroblast, and glandular cancer cell phenotypes that emerge over time. Time-course scRNA-seq measurements show that tumoroid formation activates fibroblasts, altering the extracellular matrix (ECM) composition and inducing cancer cell signal-response signatures and metabolic state change. Comparison between tumoroids with normal or cancer associated fibroblasts (CAFs) reveals different ECM compositions, as well as differential effects on cancer cell behaviors and metabolism. We identify Syndecan 1 (SDC1) and Peroxisome proliferator-activated receptor gamma (PPARG) as receptor and metabolic nodes involved in cancer cell response to CAF signals, and blocking SDC1 disrupts cancer cell growth within the tumoroid. Tumoroids from multiple PDAC patients revealed co-existence of subpopulations associated with classical and basal phenotypes, and CAF-induced migration behaviors emerged in certain patient tumoroids. Comparisons between patient tumoroids revealed a multigene migration signature that develops over time reflecting a stress response mechanism that correlates with worse clinical outcome. Altogether, stromal tumoroids can be used to explore dynamic and reciprocal interactions between cancer, CAF and endothelial cell states, and our data provides new inroads into the discovery of personalized pancreatic cancer therapies.