Viscoelasticity has emerged as a crucial design parameter for cell scaffold materials due to its governing role in key cellular processes such as differentiation, proliferation, and matrix secretion. Though this mechanosensory effect is often attributed to the direct biophysical interactions between the cells and the hydrogel, it is also known that cells can secrete their own extracellular matrix (ECM) at the cell-hydrogel interface over time, which may mediate this interaction in an unexplored manner. Here, we demonstrate that the cell-secreted ECMs can directly interact with the hydrogel scaffolds, and act as cellular transducers of hydrogel viscoelasticity. By encapsulating matrix-secreting bovine chondrocytes in hydrogels with tunable viscoelastic properties, we show that fast stress-relaxing hydrogels facilitate the interpenetration of the cell-secreted ECMs into the hydrogel scaffold, while slower stress-relaxing and elastic hydrogels prevent the interpenetration and instead cause accumulation of cell-secreted ECMs at the cell-hydrogel interface. We show that this ECM accumulation causes cellular confinement through solid stress, and drives the chondrocytes towards catabolic phenotypes observed in osteoarthritis. These findings have direct implications for designing hydrogel scaffolds for tissue engineering, and provide broad insight into chondrocyte biology under solid stress.